Facing hard problem to debug: any tips?

Development
1

Hi folks,

I’m facing a very hard problem to debug recently, which seems very random and can’t get (with the debugger I have, gdb on Windows) the full backtrace of what’s going on in the behind.

Basically, here’s a block of code section in the process():

ParamQuantity *pqPREParam9 = pTargetModule->paramQuantities[9];
if (pqPREParam9) {
	if (pqPREParam9->getValue() != 0.4f) {
		DEBUG("changed PRE");
	}
}

pqParam11->setValue(newParamValue); // pqParam11 is pTargetModule->paramQuantities[11]

ParamQuantity *pqPOSTParam9 = pTargetModule->paramQuantities[9];
if (pqPOSTParam9) {
	if (pqPOSTParam9->getValue() != 0.4f) {
		DEBUG("changed POST");
	}
}

Sometimes, the DEBUG will trig the line DEBUG(“changed POST”);

So it seems that between read param 9 pre/post and set a new value for param 11, it change the values of param 9 (and 99% of times it set the value of newParamValue, which is WEIRD.

Tried to breakpoint on widget/guy, and so on, but it seems all correct.

Maybe some other thread with some memory invasion that will replace data in the background? How would you debug this kind of stuff? Backtrace on gdb give to me only the stack of “main” function, basically:

#1  0x00007ffb9e747014 in derozer::mymodule::MyModule::process (this=0x52d1a50, args=...) at src/modules/MyModule.cpp:321
#2  0x00007ffba3d04d24 in rack::engine::Module::doProcess (this=0x52d1a50, args=...) at src/engine/Module.cpp:345
#3  0x00007ffba3d00f71 in rack::engine::Engine_stepWorker (that=0x51ac100, threadId=0) at src/engine/Engine.cpp:331
#4  rack::engine::Engine_stepFrame (that=<optimized out>) at src/engine/Engine.cpp:409
#5  rack::engine::Engine::stepBlock (this=<optimized out>, this@entry=0x4f97580, frames=<optimized out>, frames@entry=735) at src/engine/Engine.cpp:567
#6  0x00007ffba3d0173b in rack::engine::Engine_fallbackRun (that=0x4f97580) at src/engine/Engine.cpp:1349
#7  0x00007ffc3177387f in ?? () from C:\msys64\mingw64\bin\libstdc++-6.dll
#8  0x00007ffc3dd54dcb in ?? () from C:\msys64\mingw64\bin\libwinpthread-1.dll
#9  0x00007ffc517ae634 in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll
#10 0x00007ffc517ae70c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll
#11 0x00007ffc516b257d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#12 0x00007ffc52e8aa58 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()

so I can’t see what’s happens in between pqParam11->setValue(newParamValue);. I think some other thread that will fill memory with garbage? Rack API seems stable.

Note: this doesn’t always happens after setValue, sometimes few lines later on the same function. So its not directly linked to setValue

This is making me crazy.

So I’m reasoing in some options:

  1. Which actions would you do for this kind of situation?
  2. Is there any tools that can list the last 100 functions call (for example) till reaching a breakpoint?
  3. which thread Rack use expect DSP and GUI ones? I don’t think on DSP thread somethings can happens in between a module’s process() function, and on GUI side all param value for that paramid=9 seems correct on step/draw/onX methods. Are there any other thread? Otherwise, it seems another process could insert garbage? (which would be even more weird).

Thanks for any useful tips you can give it to me.

2

  • Comparing floating-point values for equality is generally not going to work. (Someone mentioned this to you in another post).

  • Rack has quite a few threads. Gui, audio, rtaudio, rtmidi are a few. Most of your code runs in either the gui thread or an audio thread. In the menus, you can choose how many processing threads. To make debugging simpler you should probably set this to 1.

  • Rack main will always be on the stack, unless you are in a callback from a service thread like rtmidi (a case which I doubt arises in your module).

  • Looks like your module is reaching into another module’s state during process. This isn’t thread safe. That module’s process may be running on another thread. That’s why expanders use the double buffer messaging to communicate.

3

Already set 1:

image

process() of every module in the patch seems to run on the same block-function:

from 0 to modulesLen. How process() of different module can run in parallel? Is not dispatch by different thread for what I see. Am I missing somethings?

4

It depends on how many thread you select. The default is one process thread.

5

So basically the whole PackOne’s CVMap module made by @stoermelder:

its basically non-thread-safe and so non-reliable? I always understand we can use get/set ParamQuantities between modules (also over the ones not made by the owner, where expander buffer is not usable).

Wrong assumption?

6

Modifying a parameter from the GUI is thread safe, obviously. I expect this code is, too. You can’t set the value directly from a different thread, but as long as you do it like the GUI does it is explicitly thread safe.

In any case, it sounds like you found your culprit. Don’t try to debug in this situation, unless you really need to.

7

Since I’m using 1 thread for debugging, this doesn’t explain why randomly the param of another module change. So no: I need to understand why this could happens.

If I move the whole logic into the GUI thread, I lost the modulation at audio-rate (neither at control rate, since the rate is not deterministic).

8

Oh, I thought from the above that you had found that is was another module changing your module’s param. If that’s not the case, I apologize for my hasty reading.

I wasn’t suggesting that you move the param stuff to to the GUI thread, I was just pointing out that it is safe and legal to to modify params from different threads, if done properly. Contrary to what was claimed.

9

Nope. pqParam11 is pTargetModule->paramQuantities[11].

So change param 11 of target module change param 9 of target module. But I don’t think its the setValue the problem: somewhere, the array get that value randomly.

I think the only way is a full backtrace…

Notice that GUI thread just call setValue on onDragMove for edit Module param, as I’m doing on DSP thread, using directly:

module->paramQuantities[paramId]

and no buffer at all. So I believe that’s feasible also on DSP thread from different thread @pachde , am I wrong?

10

Is your code run on the ui thread or the audio thread?

The ui thread can modify Param quantities

And even with thread 1 that’s just engine threads. With thread 1 rack still has 2 logical threads

2 Likes
11

Audio thread…

I hope also on audio thread we can do it right?

12

What I’m not quite understanding in the threads here is what the top level experience is intended to be. It’s one thing to be working entirely within an instance of one module. Here Rack has a pretty well-defined design for how to do things.

It’s a whole other thing if you’re talking about one module reaching into the internals of another module that you are not the author of and modifying its internals (be they params or anything else).

The design for Rack is that to make one module drive another module, it is via (virtual) voltage over cables, or by cooperating modules using the extender mechanism.

A module is free to have coupling between its params and have one param affect any property of any other param within that module. it’s possible that is happening in this case (no way to know, because we don’t know what actual modules we’re talking about here.

So, @Derozer, can you give us a higher-level description about what it is you’re trying to do?

13

Somethings similar to CVMap cited above, with differenti workflow, settings, options and such, which will fit some need I would like to have.

14

“I want to build a very advanced, complicated module. But I don’t known how”. Yes?

1 Like
15

And if thats true, should this discourage me instead of try, learn and accept che challenge? Note that CVMap does it directly on process without any sofistication, and I believe those module are pretty serious.

Anyway, the topic has nothing to do with that.

Just ask some tips for a better debugging, or a way to improve it.

16

Obviously I don’t know what is happing in your code, but you have to notice your process-method is on top of the call stack. I guess you are using an invalid pointer at some point or the call stack gets corrupted somehow.

You can assume calling setValue (or setScaledValue) from an audio thread is thread-safe as it is done in Core’s MidiMap the same way.
https://github.com/VCVRack/Rack/blob/8c6f41b778b4bf8860b89b36d5503fd37924077f/src/core/MIDIMap.cpp#L127

1 Like
17

Exactly what I Guess, thanks for the confirm.

Yes, its what is all about this thread. Dunno is the pointer. Call stack corrupted: such as? Example?

18

Very hard to say anything helpful without looking at the code. Can you show us your process?

19

Thread safe doesn’t mean unmodified on other threads though right? The issue here is thst you set and later get and the value is changed. I’m not sure why that’s a problem though. In multi threaded code it happens all the time if you aren’t locked or atomic,

I guess what’s the actual problem with the original code?

20

The problem is that nobody is changing that param 9 value, neither GUI or other actors.

It changes randonly without any external interactions (sorry, I should specify this in the main thread :P).

I see moving it just looking at the screen hehe