Hi all! Happy new year!!!

I have a question...

If I allocate memory with malloc() using processor X in a shared memory
computer, MUST I necessarily deallocate that memory using free() in the
same processor X or can I make it using processor Y. (X!=Y)
Regards,

Diego

--

From:	Dave Cronk
Subject:	Re: Allocating memory in a shared memory environment
Date:	13 Jan 2005 11:08:44 -0800

Randy wrote:
> Nick Maclaren wrote:
> > In article , Randy
wrote:
> >>Diego Andres Alvarez Marin wrote:
> >>>If I allocate memory with malloc() using processor X in a shared memory
> >>>computer, MUST I necessarily deallocate that memory using free() in the
> >>>same processor X or can I make it using processor Y. (X!=Y)
^^^^^^^^^^^^ ^^^^^^^^^^^^^^^

> Not the issue at all. Nobody asked about processors, only processes.

Not to jump in the middle of ya'll's argument (OK, I guess that is not
true), but Randy, you may want to re-read the OP before you make
statements like this.

Dave.
>
> I'll refrain from posting your grade, Nick, because that would be rude.
>
> Randy
> --
> Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--

From:	Randy
Subject:	Re: Allocating memory in a shared memory environment
Date:	13 Jan 2005 13:55:32 -0800

Dave Cronk wrote:
> Randy wrote:
>> Nick Maclaren wrote:
>>>In article , Randy wrote:
>>>>Diego Andres Alvarez Marin wrote:
>>>>>If I allocate memory with malloc() using processor X in a shared memory
>>>>>computer, MUST I necessarily deallocate that memory using free() in the
>>>>>same processor X or can I make it using processor Y. (X!=Y)
>>>>> ^^^^^^^^^^^ ^^^^^^^^^^^
>>> [Maclaren comment elided]
>>>
>>Not the issue at all. Nobody asked about processors, only processes.
>
> Not to jump in the middle of ya'll's argument (OK, I guess that is not
> true), but Randy, you may want to re-read the OP before you make
> statements like this.
>
> Dave.

Yeah. You're right. The OP did use the term "processor". I was trying to
point out that since programmers have essentially no control over processor
choice on SMPs, they instead must instead deal in processes. So while my choice
of words may be defensible, they were also unnecessarily curt. My mistake.

I guess Nick just brings out the worst in me. Certitude breeds certitude.

Randy

--
Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--

From:	russell kym horsell
Subject:	Re: Allocating memory in a shared memory environment
Date:	5 Jan 2005 22:50:55 -0800

Diego Andres Alvarez Marin wrote:
> If I allocate memory with malloc() using processor X in a shared memory
> computer, MUST I necessarily deallocate that memory using free() in the
> same processor X or can I make it using processor Y. (X!=Y)

Almost certainly: Yes.

Generally, malloc() doesn't allocate shared memory, but memory local
to a given processing node. The address will not mean anything to any
other node or process on the same node.

Again, generally shared memory (even on shared-mem processors ;-)
is allocated in UNIX with things like mmap or shmget.

--
------------ And now a word from our sponsor ------------------
Want to have instant messaging, and chat rooms, and discussion
groups for your local users or business, you need dbabble!
-- See http://netwinsite.com/sponsor/sponsor_dbabble.htm ----

From:	Randy
Subject:	Re: Allocating memory in a shared memory environment
Date:	5 Jan 2005 22:49:41 -0800

Diego Andres Alvarez Marin wrote:
> If I allocate memory with malloc() using processor X in a shared memory
> computer, MUST I necessarily deallocate that memory using free() in the
> same processor X or can I make it using processor Y. (X!=Y)

When using Unix processes or POSIX threads, you can free a pointer using any
process. All forked processes and threads share the same heap.

Randy

--
Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--
------------ And now a word from our sponsor ------------------
For a quality usenet news server, try DNEWS, easy to install,
fast, efficient and reliable. For home servers or carrier class
installations with millions of users it will allow you to grow!
---- See http://netwinsite.com/sponsor/sponsor_dnews.htm ----

From:	Nick Maclaren
Subject:	Re: Allocating memory in a shared memory environment
Date:	6 Jan 2005 22:45:58 -0800

In article , Randy wrote:
>Diego Andres Alvarez Marin wrote:
>> If I allocate memory with malloc() using processor X in a shared memory
>> computer, MUST I necessarily deallocate that memory using free() in the
>> same processor X or can I make it using processor Y. (X!=Y)
>
>When using Unix processes or POSIX threads, you can free a pointer using any
>process. All forked processes and threads share the same heap.

0/10. Despite having got the right one-word answer ("yes").

Each Unix/POSIX process uses a separate data segment (a.k.a. 'heap'
as used by malloc and friends), but each thread within a process
shared the same one. There can be both thread-specific and process-
generic 'heaps', but they are not used by malloc.

The binding of processes and threads to processORs (i.e. CPUs) is
a very complicated issue. The simple answer is that it may be
ignored as far as the semantics of most normal memory allocation is
concerned.

Regards,
Nick Maclaren.

--
------------ And now a word from our sponsor ----------------------
For a quality mail server, try SurgeMail, easy to install,
fast, efficient and reliable. Run a million users on a standard
PC running NT or Unix without running out of power, use the best!
---- See http://netwinsite.com/sponsor/sponsor_surgemail.htm ----

From:	Randy
Subject:	Re: Allocating memory in a shared memory environment
Date:	9 Jan 2005 21:52:47 -0800

Nick Maclaren wrote:
> In article , Randy wrote:
>>Diego Andres Alvarez Marin wrote:
>>>If I allocate memory with malloc() using processor X in a shared memory
>>>computer, MUST I necessarily deallocate that memory using free() in the
>>>same processor X or can I make it using processor Y. (X!=Y)
>>
>>When using Unix processes or POSIX threads, you can free a pointer using any
>>process. All forked processes and threads share the same heap.
>
> 0/10. Despite having got the right one-word answer ("yes").
>
> Each Unix/POSIX process uses a separate data segment (a.k.a. 'heap'
> as used by malloc and friends), but each thread within a process
> shared the same one. There can be both thread-specific and process-
> generic 'heaps', but they are not used by malloc.

To verify your commentary, write a short code with one thread mallocing into a
global pointer and the other reading the malloced data via that pointer. You'll
find that both processes share the same heap and the same global data. That
they share the same heap was precisely what the OP was asking. As such, either
process can free the allocated data via the shared pointer.

Likewise, both share the same (global) data segment, or one process could never
share data with another process. It's only the process' stacks that are
distinct (which is also not what he was asking).

BTW, "data segment" and "heap" are definitely NOT the same beast.

> The binding of processes and threads to processORs (i.e. CPUs) is
> a very complicated issue. The simple answer is that it may be
> ignored as far as the semantics of most normal memory allocation is
> concerned.

Not the issue at all. Nobody asked about processors, only processes.

I'll refrain from posting your grade, Nick, because that would be rude.

Randy
--
Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--

From:	russell kym horsell
Subject:	Re: Allocating memory in a shared memory environment
Date:	12 Jan 2005 01:01:08 -0800

Randy wrote:
....
> To verify your commentary, write a short code with one thread mallocing into a
> global pointer and the other reading the malloced data via that pointer. You'll
> find that both processes share the same heap and the same global data. That
> they share the same heap was precisely what the OP was asking. As such, either
> process can free the allocated data via the shared pointer.

[In case someone is reading this: ;-)]
Only if they're "related by birth", of course.
It'd be neat security if 2 random processes shared the same heap.

Back to the original Q.

It is certainly tru that in (e.g.) an SMP environment where
a given ap forks off children we have a "shared memory environement"
where each processes shares the same heap.

But in typical shared-mem aps from the not-too-distant past, processes
on different nodes shared memory via some mechianism like mmap
and could become quite unhaoppy if a process on one node mallocated
a chunk, planted the pointed to it in shared mem, and expected a process
on another node to be able to free is up again.

Haven't we all maintained parallel code like that? ;-)

--
------------ And now a word from our sponsor ------------------
For a quality usenet news server, try DNEWS, easy to install,
fast, efficient and reliable. For home servers or carrier class
installations with millions of users it will allow you to grow!
---- See http://netwinsite.com/sponsor/sponsor_dnews.htm ----

From:	Nick Maclaren
Subject:	Re: Allocating memory in a shared memory environment
Date:	12 Jan 2005 00:54:53 -0800

In article ,
Randy writes:
|> > Each Unix/POSIX process uses a separate data segment (a.k.a. 'heap'
|> > as used by malloc and friends), but each thread within a process
|> > shared the same one. There can be both thread-specific and process-
|> > generic 'heaps', but they are not used by malloc.
|>
|> To verify your commentary, write a short code with one thread mallocing into a
|> global pointer and the other reading the malloced data via that pointer. You'll
|> find that both processes share the same heap and the same global data. That
|> they share the same heap was precisely what the OP was asking. As such, either
|> process can free the allocated data via the shared pointer.

Bizarre. Since I am too kind for my own good, here is a program for you:

#include
#include
#include

int main (void) {
int *d = malloc(sizeof(int));
pid_t p;

*d = 0;
if (p = fork()) {
printf("%d %d\n",p,*d);
*d = 11;
sleep(5);
printf("%d %d\n",p,*d);
*d = 12;
sleep(10);
printf("%d %d\n",p,*d);
} else {
printf("%d %d\n",p,*d);
*d = 21;
sleep(10);
printf("%d %d\n",p,*d);
*d = 22;
sleep(10);
printf("%d %d\n",p,*d);
}
return 0;
}

|> Likewise, both share the same (global) data segment, or one process
|> could never share data with another process. It's only the process'
|> stacks that are distinct (which is also not what he was asking).
|>
|> BTW, "data segment" and "heap" are definitely NOT the same beast.

You are very confused. The primary data segment on a modern Unix
(or POSIX) system is very often implemented as a memory mapped
segment, but it is always a PRIVATE one. It is only SHARED data
segments that are shared (not surprisingly). PRIVATE ones are copied
by fork (sometimes just by flagging as copy-on-write), but thereafter
are distinct.

While I am aware of many distinctions between data segments and heaps,
in the Unix context and otherwise, the primary data segment (i.e. the
one used by malloc and friends) has been called the heap by many Unix
people for at least 25 years.

Regards,
Nick Maclaren.

--
------------ And now a word from our sponsor ---------------------
For a secure high performance FTP using SSL/TLS encryption
upgrade to SurgeFTP
---- See http://netwinsite.com/sponsor/sponsor_surgeftp.htm ----

From:	Randy
Subject:	Re: Allocating memory in a shared memory environment
Date:	12 Jan 2005 20:51:46 -0800

Nick Maclaren wrote:
> In article ,
> Randy writes:
....
> |>
> |> To verify your commentary, write a short code with one thread mallocing into a
> |> global pointer and the other reading the malloced data via that pointer. You'll
> |> find that both processes share the same heap and the same global data. That
> |> they share the same heap was precisely what the OP was asking. As such, either
> |> process can free the allocated data via the shared pointer.
>
> Bizarre. Since I am too kind for my own good, here is a program for you:

What fun! Dueling process models!

Here's the output from your program (using heavyweight threads):

0 0
21940 0
21940 11
0 21
21940 12
0 22

And because you are "too kind", so must I be also. Here's an equivalent program
using pthreads that behaves differently from yours, yet acts as I said it would:

"
#include
#include
#include
#include

static int *d;
pthread_t thread1, thread2;

void * proc1( void*s)
{
printf( "%s d addr: %p\n", s, d);
printf("%s %d\n", s,*d);
*d = 11;
sleep(5);
printf("%s %d\n", s,*d);
*d = 12;
sleep(10);
printf("%s %d\n", s,*d);
sleep(7);
}

void * proc2( void*s)
{
printf( "%s d addr: %p\n", s, d);
printf("%s %d\n", s,*d);
*d = 21;
sleep(10);
printf("%s %d\n", s,*d);
*d = 22;
sleep(10);
printf("%s %d\n", s,*d);
}

int main (void) {
d = malloc(sizeof(int));
*d = 0;
pthread_create( &thread1, NULL, proc1, (void*)"thread1");
pthread_create( &thread2, NULL, proc2, (void*)"thread2");
pthread_join( thread1, NULL);
pthread_join( thread2, NULL);

return 0;
}
"

And here's its output (using lightweight threads):

thread1 d addr: 0x804adb8
thread1 0
thread2 d addr: 0x804adb8
thread2 11 # what's this?
thread1 21 # oh dear!
thread2 12 # encroyable!
thread1 22 # no, no, no...
thread2 22 # hmm

You'll note that thread2's heap value is changed by thread1 and vice versa.
Ergo the processes share the same heap. You'll also note that they share the
same global data, since global variable "d" resolves to the same address, and
each process clearly uses it to interfere with the same malloced value.

What does this mean, you ask? Why would Linux's heavyweight processes share
data differently than its lightweight processes? Frankly, I don't know. I've
done very little programming using heavyweight processes. My experience lies
with pthreads and OpenMP, which both obey the model of heap sharing that I describe.

The only way I can interpret the results of your code is if fork() spawned two
completely independent executables with entirely independent data segments and
heaps and shared no state at all, except having identical address spaces. But
AFAIK, that's not what's supposed to happen in Unix. Global/static/data segment
variables are *supposed* to be shared by parent and child processes.

Interestingly, if I move your "d" variable outside main and make it a
non-pointer integer, your program still produces the same output as before, even
without a heap. The two processes apparently have distinct values for "d".
Clearly, child processes in Linux do not share ANY data with their parent
process. Weird.

But clearly, your code's "d" variable (when global) is not being shared among
the processes. Therefore, while it resides at the same address in both
processes, it cannot access the same heap, and so, it fails to test your
hypothesis that the two process's heaps are distinct.

....
> |>
> |> BTW, "data segment" and "heap" are definitely NOT the same beast.
>
> You are very confused.

Clearly I am, but not in the way you think...

> The primary data segment on a modern Unix
> (or POSIX) system is very often implemented as a memory mapped
> segment, but it is always a PRIVATE one. It is only SHARED data
> segments that are shared (not surprisingly). PRIVATE ones are copied
> by fork (sometimes just by flagging as copy-on-write), but thereafter
> are distinct.
>
> While I am aware of many distinctions between data segments and heaps,
> in the Unix context and otherwise, the primary data segment (i.e. the
> one used by malloc and friends) has been called the heap by many Unix
> people for at least 25 years.

News to me. For every executable I know (Unix, Windows, and DOS), its "data"
segment (nm labels these symbols with a B or D, respectively part of BSS or DSS)
is static, it physically exists on the disk prior to being loaded in situ into
memory, and the DSS segment is initialized to zero. Perhaps there's a more
appropriate, more standard term for a process's global data section/segment than
..data, but I don't know what it is.

The heap is dynamic, does not exist in the disk image of the executable, is
never initialized, and on every system I've used, it's shared among all
processes that share the same parent process. Not so for heavyweight processes,
however. Again, I can't explain what Linux is doing in your program.

BTW, the GNU binutils assembler refers to global variables as being in .comm.
According to the following URL, variables in .comm will map into the ".data
section" of the program:

http://publibn.boulder.ibm.com/doc_link/en_US/a_doc_lib/aixassem/alangref/comm.htm

Is the "data section" different from the "data segment"? Hell, I don't know.

Randy

--
Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--
------------ And now a word from our sponsor ---------------------
For a secure high performance FTP using SSL/TLS encryption
upgrade to SurgeFTP
---- See http://netwinsite.com/sponsor/sponsor_surgeftp.htm ----

From:	Dale Talcott
Subject:	Re: Allocating memory in a shared memory environment
Date:	14 Jan 2005 10:03:04 -0800

Randy writes:

>You'll note that thread2's heap value is changed by thread1 and vice versa.
>Ergo the processes share the same heap. You'll also note that they share the
>same global data, since global variable "d" resolves to the same address, and
>each process clearly uses it to interfere with the same malloced value.

>What does this mean, you ask? Why would Linux's heavyweight processes share
>data differently than its lightweight processes? Frankly, I don't know. I've
>done very little programming using heavyweight processes. My experience lies
>with pthreads and OpenMP, which both obey the model of heap sharing that I describe.

>The only way I can interpret the results of your code is if fork() spawned two
>completely independent executables with entirely independent data segments and
>heaps and shared no state at all, except having identical address spaces. But
>AFAIK, that's not what's supposed to happen in Unix. Global/static/data segment
>variables are *supposed* to be shared by parent and child processes.

No. In the Unix and Linux worlds, fork() creates entirely separate address
spaces. This isn't quite true, in that most Unixes implement a fork by
setting all memory regions to copy-on-write, then point both parent and
child at the same regions. So long as neither process modifies a page
of memory, they share it. As soon as one process modifies a page, it
gets a private copy of the page and the other process keeps using the
original version. The result is that there is no way for parent and
child to communicate values through memory. [I'm ignoring the explicit
shared memory operations for now.]

>News to me. For every executable I know (Unix, Windows, and DOS), its "data"
>segment (nm labels these symbols with a B or D, respectively part of BSS or DSS)
>is static, it physically exists on the disk prior to being loaded in situ into
>memory, and the DSS segment is initialized to zero.

The model I am familiar with is a little more complicated. The data
segment comes in two parts - initialized (DSS) and uninitialized (BSS).
The initialized part is loaded directly from disk, but might be patched
right after loading to fix pointers to shared objects.

The uninitialized portion is defined on disk simply by a size. It
comes after the initialized portion and is set to zero. The heap
usually starts at the end of the BSS portion.

>The heap is dynamic, does not exist in the disk image of the executable, is
>never initialized,

Actually, on Linux and Unix, the heap is initialized to zero once, but
only because every new memory page added to a process's address space
is set to zero initially.

>and on every system I've used, it's shared among all
>processes that share the same parent process.

Not so on Linux and Unix, as explained above.

Now, some confusion might have originated from the early implementation
of threads under Linux. My understanding is that up to kernel 2.6, when
you created a new thread, Linux actually created a new process, but with
exactly the same memory map as the process creating the thread. So,
here you had the case where two processes were sharing read and write
access to the same data areas. These thread-processes weren't full
processes and had a tighter parent-child relationship than processes
created via fork.

I am not an expert on Linux threads, so corrections are welcome.

--
Dale Talcott, Rosen Center for Advanced Computing, Purdue University
aeh@quest.cc.purdue.edu http://quest.cc.purdue.edu/~aeh/

--
------------ And now a word from our sponsor ------------------
Want to have instant messaging, and chat rooms, and discussion
groups for your local users or business, you need dbabble!
-- See http://netwinsite.com/sponsor/sponsor_dbabble.htm ----

From:	Randy
Subject:	Re: Allocating memory in a shared memory environment
Date:	14 Jan 2005 13:51:34 -0800

Dale Talcott wrote:
....
> No. In the Unix and Linux worlds, fork() creates entirely separate address
> spaces. This isn't quite true, in that most Unixes implement a fork by
> setting all memory regions to copy-on-write, then point both parent and
> child at the same regions. So long as neither process modifies a page
> of memory, they share it. As soon as one process modifies a page, it
> gets a private copy of the page and the other process keeps using the
> original version. The result is that there is no way for parent and
> child to communicate values through memory. [I'm ignoring the explicit
> shared memory operations for now.]
....
>
> Now, some confusion might have originated from the early implementation
> of threads under Linux. My understanding is that up to kernel 2.6, when
> you created a new thread, Linux actually created a new process, but with
> exactly the same memory map as the process creating the thread. So,
> here you had the case where two processes were sharing read and write
> access to the same data areas. These thread-processes weren't full
> processes and had a tighter parent-child relationship than processes
> created via fork.

OK. If I may summarize then the semantic differences between doing a fork and
spawning a thread:

Fork:

The child process gets a clone of the parent process's global data, stack, and
heap. No data space is shared. Subsequent changes to data by each process
affect only that process's data. Only the originating process can free malloced
memory. (Although, presumably, child processes can free their copy of heap data
that was originally malloced by the parent.)

Thread:

Child threads share the parent's data, stack, and heap. All data space is
shared (except perhaps the data lying in the parent process' original stack).
Subsequent changes to global data or heap affect all threads spawned from that
parent (as well as the parent), although no changes to new stack data are
shared. Any thread (or the parent process) can free malloced memory. I'm
pretty sure these semantics have remained consistent throughout Linux's
evolution, although its implementation of lightweight processes differs between
its current NPTL (LWP) and its old threading model (largely HWP).

Although the OP *might* be running forked HWPs on multiple processors, I suspect
he's using LWPs (pthreads/OpenMP), in which case, the thread semantics should
answer the original malloc/heap question.

Thanks for the edification.

Randy

--
Randy Crawford http://www.ruf.rice.edu/~rand rand AT rice DOT edu

--

From:	Nick Maclaren
Subject:	Re: Allocating memory in a shared memory environment
Date:	16 Jan 2005 21:55:45 -0800

In article , Randy wrote:
>OK. If I may summarize then the semantic differences between doing a fork and
>spawning a thread:

Congratulations. This is still wrong in places, but is a lot better.
The example code that you posted shows that you are still seriously
confused about the difference between a process and a thread - you
should find out about that before investigating what happens with
memory, as the process/thread distinction is critical to that.

>Fork:
>
>The child process gets a clone of the parent process's global data, stack, and
>heap. No data space is shared. Subsequent changes to data by each process
>affect only that process's data. Only the originating process can free malloced
>memory. (Although, presumably, child processes can free their copy of heap data
>that was originally malloced by the parent.)

Correct as far as the local data segments are concerned (including,
here, shared data segments allocated PRIVATE). There is no "presumably"
about it - that is how it is required to work.

But wrong as far as shared memory segments are concerned (there is no
such thing as global data in the Unix or POSIX model). They ARE shared
over fork, too, though there were some pre-POSIX interfaces where they
were not. The text is:

Memory mappings created in the parent are retained in the child
process. MAP_PRIVATE mappings inherited from the parent shall also
be MAP_PRIVATE mappings in the child, and any modifications to the
data in these mappings made by the parent prior to calling fork( )
shall be visible to the child. Any modifications to the data in
MAP_PRIVATE mappings made by the parent after fork( ) returns
shall be visible only to the parent. Modifications to the data in
MAP_PRIVATE mappings made by the child shall be visible only to
the child.

>Thread:
>
>Child threads share the parent's data, stack, and heap. All data space is
>shared (except perhaps the data lying in the parent process' original stack).
>Subsequent changes to global data or heap affect all threads spawned from that
>parent (as well as the parent), although no changes to new stack data are
>shared. Any thread (or the parent process) can free malloced memory. I'm
>pretty sure these semantics have remained consistent throughout Linux's
>evolution, although its implementation of lightweight processes differs between
>its current NPTL (LWP) and its old threading model (largely HWP).

Except for the description of the stack, this is largely right. It has
not been true of all threading models, but has been so in all of the
ones that POSIX was derived from or has led to, as far as I know. With
the exeption of a few explicitly thread-specific operations, ANY change
to the environment may be done by any thread.

[Aside: people with experience of IBM MVT tasking and other highly
non-POSIX models may remember some interesting problems when attempting
to implement POSIX-style threading on them. For example, you could
read and write data from any thread, but could close or extend a file
only from the thread that had opened it. As extension was done behind
the scenes when you ran out of space, this was, er, a trap for the
unwary.]

However, back to the stacks (and thread-private 'heap' data, for that
matter). POSIX is unclear on this, but everyone assumes that they can
be allocated and freed only by their owning thread, but may be accessed
from any thread. I.e. if one thread passes a pointer to its local
stack or private data to another thread, the latter can update it
normally. If anyone has found an explicit statement in POSIX that
confirms this is the case (or that it is not), I should like to hear.

>Although the OP *might* be running forked HWPs on multiple processors, I suspect
>he's using LWPs (pthreads/OpenMP), in which case, the thread semantics should
>answer the original malloc/heap question.

The processor difference is irrelevant. Not merely can you run each
thread of a process on a separate processor, you can run multiple
processes on a single processor. I have managed systems with a single
Unix image on distributed memory processors, and have some experience
of distributed shared memory interfaces. But those are MUCH more
advanced issues than are relevant to this thread.

Where I came in was your posting where you described the same memory
sharing semantics for processes and threads, which is not so. The
above may help.

Regards,
Nick Maclaren.

--