Concurrency Best Practices + Tests
Hi everyone, Can I get your input on the following questions : - What are your best practices and recommendations for developing and testing concurrent software? - How to discover need for synchronization/critical sections/ when doing TDD? - How to write code to avoid dead-locks? Noury
On Wed, 4 Sep 2019 at 21:32, Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
I haven't any formalized advice, but just a scattering of thoughts: - When the debugger opens, it really disturbs the timing context between threads, so the problem you are looking for disappears. - Tracking down timing-related problems often required interleaved-threads logging to a threadsafe queue so a historical sequence could be reviewed. Sometimes this included exporting it to Excel then manually chopping repeating sectioning into side-by-side columns to see how the order of execution differs between cycles. - Correctly distinguish between your "shared resource" problems and your "work signalling" producer/consumer problems, respectively using mutexes and semaphores. I think the latter is easier to reason about. btw, watch out for Announcements. You will appear to register them for the receiving-thread, but actually they are executed in the context of the sending-thread. - How to discover need for synchronization/critical sections/ when doing
TDD?
First consider whether synchronisation can be avoided using queues with "positive hand-off" http://www.flounder.com/no_synchronization.htm This video I bumped into today touches on the same thing... https://www.youtube.com/watch?v=2yXtZ8x7TXw Not available right now, but along the line "apartment-threading" I have been musing about VM support for "Process-specific-immutability" So an object has an "owner-process" which is the only thread able to mutate it. The existing immutability flag would be turned on, and when triggered the VM would check if the thread-id matched before allowing/blocking an update. For example, in a producer/consumer arrangement there could be a specific system thread dedicated to processing class/code-modifications which could keep that safe without scattering mutexes through that code, but anyone can read it. One thing that wasn't obvious to me until I read it, was that lazy-initialization which is common in TDD is problematic with concurrency. I'm still mulling over the implications, but it seems reasonable that you can get at least double-initialization. So its better to initialize at creation time before it is handed around to other threads. - How to write code to avoid dead-locks?
Here are some bookmarked articles I found enlightening on the differences between between mutexes and semaphores. The last two identify typical problems. 1. https://barrgroup.com/Embedded-Systems/How-To/RTOS-Mutex-Semaphore 2. http://blog.feabhas.com/2009/09/mutex-vs-semaphores-%e2%80%93-part-1-semapho... 3. http://blog.feabhas.com/2009/09/mutex-vs-semaphores-%e2%80%93-part-2-the-mut... 4. http://blog.feabhas.com/2009/10/mutex-vs-semaphores-%e2%80%93-part-3-final-p... 5. http://www.smxrtos.com/articles/techppr/mutex.htm You may find some useful patterns in these books... - http://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf - https://pdfs.semanticscholar.org/a51b/40c904add79f590ed887ccd3a78dbe630938.p... cheers -ben
Noury - I happened to notice in a recent article about the Rust scheduler (it caught my eye) it had a section on concurrent testing and a tool they write called Loom to test all possible permutations and catch errors. This might be an avenue of investigation for your work ? An idea any way. https://tokio.rs/blog/2019-10-scheduler/ Sent from my iPhone
On 4 Sep 2019, at 14:31, Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
Hello, My answer is late, but hopefully still useful. I'm not a concurrent programmer in Pharo, but have done it in lots of other environments. Use the immutable object pattern to avoid having to deal with race conditions (although it won't solve all the problems). POSA2 was focused on OO concurrency, and had been applied in C++ and Java: http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/ <http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/> You can find Coursera MOOCs with modern application of the POSA2 patterns, but it's mostly focused on Android. Advice is often to use an environment's thread-safe data structures and thread pool librairies, if they exist. It's a bigger challenge otherwise to create these yourself. I saw a cool exercise of understanding race conditions for the Singleton pattern in the book Head First Design Patterns. It basically looks at breaking up source code of a concurrent method into chunks, and figuring out if you can get a race condition with two threads executing them separately. The metaphor is magnets on a refrigerator door. See the attached image. The Spin model checker (using a DSL, called PROMELA, for code modeling) works in a similar way, by creating all possible executions and checking for violations of invariants. But that's a formal method that is hard to apply to arbitrary (complex) code. Cheers, Cris On Wed, Sep 4, 2019, 09:32 Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
Hi Q. What are your best practices and recommendations for developing and testing concurrent software? A. I avoid writing concurrent code if I can. If not, I would launch multiple Smalltalk instances instead. Even then I would use SQLite3 (or even Postgres, Redis etc) to hold the shared mutable state. SQlite3 locks the db for writes, which is exactly the behaviour I need to controlled access. SQlite3 and Redis can be in memory and can be very fast. I use UDBCSQLite3, but looking at Svenâs SimpleRedisClient (https://github.com/svenvc/SimpleRedisClient) out of sheer curiosity. Then, I can just write sequential code without worrying too much about race/starvation conditions. If not then it class-side mutexes and semaphores, but only as a last resort. Q. How to discover need for synchronization/critical sections/ when doing TDD? A. I do the opposite. I make everything critical, and instead look for places where I can do Process yield. Brutal, but effective. Q. How to write code to avoid dead-locks? A. Use SQLite3 to hold shared mutable resources. It locks the entire db for writes, solving the deadlock for me. I will give SimpleRedisClient a run sometime. Vince From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Christopher Fuhrman Sent: Tuesday, 15 October 2019 11:39 AM To: Any question about pharo is welcome <pharo-users@lists.pharo.org> Subject: Re: [Pharo-users] Concurrency Best Practices + Tests EXTERNAL: Do not click links or open attachments if you do not recognize the sender. Hello, My answer is late, but hopefully still useful. I'm not a concurrent programmer in Pharo, but have done it in lots of other environments. Use the immutable object pattern to avoid having to deal with race conditions (although it won't solve all the problems). POSA2 was focused on OO concurrency, and had been applied in C++ and Java: http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/<https://urldefense.com/v3/__http:/www.dre.vanderbilt.edu/*schmidt/POSA/POSA2/__;fg!09EJrJkydGw!RZmj5IufCu5nBi1-U1FHb6CpYodkamBVuIYPrJLsQ3RGIaDU5FpZeJftWaDoHGXSAoge9Uoe5w$> You can find Coursera MOOCs with modern application of the POSA2 patterns, but it's mostly focused on Android. Advice is often to use an environment's thread-safe data structures and thread pool librairies, if they exist. It's a bigger challenge otherwise to create these yourself. I saw a cool exercise of understanding race conditions for the Singleton pattern in the book Head First Design Patterns. It basically looks at breaking up source code of a concurrent method into chunks, and figuring out if you can get a race condition with two threads executing them separately. The metaphor is magnets on a refrigerator door. See the attached image. The Spin model checker (using a DSL, called PROMELA, for code modeling) works in a similar way, by creating all possible executions and checking for violations of invariants. But that's a formal method that is hard to apply to arbitrary (complex) code. Cheers, Cris On Wed, Sep 4, 2019, 09:32 Noury Bouraqadi <bouraqadi@gmail.com<mailto:bouraqadi@gmail.com>> wrote: Hi everyone, Can I get your input on the following questions : - What are your best practices and recommendations for developing and testing concurrent software? - How to discover need for synchronization/critical sections/ when doing TDD? - How to write code to avoid dead-locks? Noury
Thanks Vince, Christopher, and Tim! Noury
On 15 Oct 2019, at 05:46, Vince Refiti <vince.refiti@trapezegroup.com.au> wrote:
Hi
Q. What are your best practices and recommendations for developing and testing concurrent software? A. I avoid writing concurrent code if I can. If not, I would launch multiple Smalltalk instances instead.
Even then I would use SQLite3 (or even Postgres, Redis etc) to hold the shared mutable state. SQlite3 locks the db for writes, which is exactly the behaviour I need to controlled access.
SQlite3 and Redis can be in memory and can be very fast. I use UDBCSQLite3, but looking at Svenâs SimpleRedisClient (https://github.com/svenvc/SimpleRedisClient <https://github.com/svenvc/SimpleRedisClient>) out of sheer curiosity. Then, I can just write sequential code without worrying too much about race/starvation conditions. If not then it class-side mutexes and semaphores, but only as a last resort.
Q. How to discover need for synchronization/critical sections/ when doing TDD? A. I do the opposite. I make everything critical, and instead look for places where I can do Process yield. Brutal, but effective.
Q. How to write code to avoid dead-locks? A. Use SQLite3 to hold shared mutable resources. It locks the entire db for writes, solving the deadlock for me. I will give SimpleRedisClient a run sometime.
Vince
From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Christopher Fuhrman Sent: Tuesday, 15 October 2019 11:39 AM To: Any question about pharo is welcome <pharo-users@lists.pharo.org <mailto:pharo-users@lists.pharo.org>> Subject: Re: [Pharo-users] Concurrency Best Practices + Tests
EXTERNAL: Do not click links or open attachments if you do not recognize the sender.
Hello,
My answer is late, but hopefully still useful. I'm not a concurrent programmer in Pharo, but have done it in lots of other environments.
Use the immutable object pattern to avoid having to deal with race conditions (although it won't solve all the problems).
POSA2 was focused on OO concurrency, and had been applied in C++ and Java: http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/ <https://urldefense.com/v3/__http:/www.dre.vanderbilt.edu/*schmidt/POSA/POSA2...>
You can find Coursera MOOCs with modern application of the POSA2 patterns, but it's mostly focused on Android.
Advice is often to use an environment's thread-safe data structures and thread pool librairies, if they exist. It's a bigger challenge otherwise to create these yourself.
I saw a cool exercise of understanding race conditions for the Singleton pattern in the book Head First Design Patterns. It basically looks at breaking up source code of a concurrent method into chunks, and figuring out if you can get a race condition with two threads executing them separately. The metaphor is magnets on a refrigerator door. See the attached image. The Spin model checker (using a DSL, called PROMELA, for code modeling) works in a similar way, by creating all possible executions and checking for violations of invariants. But that's a formal method that is hard to apply to arbitrary (complex) code.
Cheers,
Cris
On Wed, Sep 4, 2019, 09:32 Noury Bouraqadi <bouraqadi@gmail.com <mailto:bouraqadi@gmail.com>> wrote: Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
Iâd love to see someone write a recording debugger for smalltalk. In theory you might be able to use rr with Pharo. https://rr-project.org/ ../Dave On Oct 15, 2019, 9:22 AM -0400, Noury Bouraqadi <bouraqadi@gmail.com>, wrote:
Thanks Vince, Christopher, and Tim!
Noury
On 15 Oct 2019, at 05:46, Vince Refiti <vince.refiti@trapezegroup.com.au> wrote:
Hi
Q. What are your best practices and recommendations for developing and testing concurrent software? A. I avoid writing concurrent code if I can. If not, I would launch multiple Smalltalk instances instead.
Even then I would use SQLite3 (or even Postgres, Redis etc) to hold the shared mutable state. SQlite3 locks the db for writes, which is exactly the behaviour I need to controlled access.
SQlite3 and Redis can be in memory and can be very fast. I use UDBCSQLite3, but looking at Svenâs SimpleRedisClient (https://github.com/svenvc/SimpleRedisClient) out of sheer curiosity. Then, I can just write sequential code without worrying too much about race/starvation conditions. If not then it class-side mutexes and semaphores, but only as a last resort.
Q. How to discover need for synchronization/critical sections/ when doing TDD? A. I do the opposite. I make everything critical, and instead look for places where I can do Process yield. Brutal, but effective.
Q. How to write code to avoid dead-locks? A. Use SQLite3 to hold shared mutable resources. It locks the entire db for writes, solving the deadlock for me. I will give SimpleRedisClient a run sometime.
Vince
From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Christopher Fuhrman Sent: Tuesday, 15 October 2019 11:39 AM To: Any question about pharo is welcome <pharo-users@lists.pharo.org> Subject: Re: [Pharo-users] Concurrency Best Practices + Tests
EXTERNAL:Â Do not click links or open attachments if you do not recognize the sender.
Hello,
My answer is late, but hopefully still useful. I'm not a concurrent programmer in Pharo, but have done it in lots of other environments.
Use the immutable object pattern to avoid having to deal with race conditions (although it won't solve all the problems).
POSA2 was focused on OO concurrency, and had been applied in C++ and Java: http://www.dre.vanderbilt.edu/~schmidt/POSA/POSA2/
You can find Coursera MOOCs with modern application of the POSA2 patterns, but it's mostly focused on Android.
Advice is often to use an environment's thread-safe data structures and thread pool librairies, if they exist. It's a bigger challenge otherwise to create these yourself.
I saw a cool exercise of understanding race conditions for the Singleton pattern in the book Head First Design Patterns. It basically looks at breaking up source code of a concurrent method into chunks, and figuring out if you can get a race condition with two threads executing them separately. The metaphor is magnets on a refrigerator door. See the attached image. The Spin model checker (using a DSL, called PROMELA, for code modeling) works in a similar way, by creating all possible executions and checking for violations of invariants. But that's a formal method that is hard to apply to arbitrary (complex) code.
Cheers,
Cris
On Wed, Sep 4, 2019, 09:32 Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
(1) Be very clear in your design about which objects are shared and which are not. (2) Immutable objects can be shared safely. Mutable ones are much much harder to work with. (3) Lazy initialisation does not work well in concurrent programming, *except* when you are constructing an immutable value and it doesn't matter if the variable is initialised twice. (4) If you can possibly do so, *SEPARATE* the concurrency structure from the application- specific work. Use something like Promela/SPIN, or NuSMV, or NuXmv, to check your concurrency structure, or CPN Tools if you prefer Coloured Petri Nets as a modelling notation. This is what Erlang does with great success with "behaviours"; the concurrency structures are there in the library and you just plug in the non-concurrent part. Look up "parallelism patterns" and "algorithmic skeletons". The book https://pdfs.semanticscholar.org/f43e/7c6a40b96743f2217472a49cd616622bdc26.p... may be helpful. Separation of concerns: this way you can test the concurrency structure of your code independently of the rest. (5) It isn't clear to me that TDD is a good way to develop concurrent programs. It's a great way to develop the non-concurrent *parts*. One reason I say this is that I've found that it is better to avoid adding unneeded sequencing in a design than to add concurrency to a sequential code. If you try to *add* critical sections in a language like Lisp or Smalltalk, you *will* miss some. So the TDD Way of is likely to seduce you into making your code too sequential too soon. (6) I personally find processes communicating via SharedQueues *much* easier to design and make work than processes communicating via shared variables and mutexes or semaphores. (Basically, a Semaphore is a SharedQueue that can only remember how many items it's holding, not what they were.) Your experience may be different, but Processes with private data communicating via SharedQueues transfer much easier to concurrent activities on separate machines communicating via sockets. On Thu, 5 Sep 2019 at 01:32, Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
I can definitely second all the points. If you want to verify your design formally, Vereofy (http://www.vereofy.de) is another model checker that focuses on communication and coordination. Best, Steffen Am .10.2019, 08:49 Uhr, schrieb Richard O'Keefe <raoknz@gmail.com>:
(1) Be very clear in your design about which objects are shared and which are not. (2) Immutable objects can be shared safely. Mutable ones are much much harder to work with. (3) Lazy initialisation does not work well in concurrent programming, *except* when you are constructing an immutable value and it doesn't matter if the variable is initialised twice. (4) If you can possibly do so, *SEPARATE* the concurrency structure from the application- specific work. Use something like Promela/SPIN, or NuSMV, or NuXmv, to check your concurrency structure, or CPN Tools if you prefer Coloured Petri Nets as a modelling notation. This is what Erlang does with great success with "behaviours"; the concurrency structures are there in the library and you just plug in the non-concurrent part. Look up "parallelism patterns" and "algorithmic skeletons". The book https://pdfs.semanticscholar.org/f43e/7c6a40b96743f2217472a49cd616622bdc26.p... may be helpful. Separation of concerns: this way you can test the concurrency structure of your code independently of the rest. (5) It isn't clear to me that TDD is a good way to develop concurrent programs. It's a great way to develop the non-concurrent *parts*. One reason I say this is that I've found that it is better to avoid adding unneeded sequencing in a design than to add concurrency to a sequential code. If you try to *add* critical sections in a language like Lisp or Smalltalk, you *will* miss some. So the TDD Way of is likely to seduce you into making your code too sequential too soon. (6) I personally find processes communicating via SharedQueues *much* easier to design and make work than processes communicating via shared variables and mutexes or semaphores. (Basically, a Semaphore is a SharedQueue that can only remember how many items it's holding, not what they were.) Your experience may be different, but Processes with private data communicating via SharedQueues transfer much easier to concurrent activities on separate machines communicating via sockets.
On Thu, 5 Sep 2019 at 01:32, Noury Bouraqadi <bouraqadi@gmail.com> wrote:
Hi everyone,
Can I get your input on the following questions :
- What are your best practices and recommendations for developing and testing concurrent software?
- How to discover need for synchronization/critical sections/ when doing TDD?
- How to write code to avoid dead-locks?
Noury
participants (8)
-
Ben Coman -
Christopher Fuhrman -
David Mason -
Noury Bouraqadi -
Richard O'Keefe -
Steffen Märcker -
Tim Mackinnon -
Vince Refiti