You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 13:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 23, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15256210187,10sleep10-21swapper/119:08:231
1421129480,9sleep30-21swapper/319:05:233
1524728164,13sleep20-21swapper/219:08:162
1507728169,8sleep00-21swapper/019:06:050
241102460,0sleep00-21swapper/022:27:280
15464992622,3cyclictest0-21swapper/122:45:141
1545799261,24cyclictest0-21swapper/022:58:340
15464992522,2cyclictest8092-21diskmemload22:55:161
15464992521,4cyclictest0-21swapper/122:20:391
15464992517,7cyclictest8092-21diskmemload23:32:381
15464992418,1cyclictest8092-21diskmemload23:36:441
15464992410,9cyclictest0-21swapper/123:40:141
1546999230,23cyclictest0-21swapper/223:14:562
1546999230,23cyclictest0-21swapper/222:44:562
1546999230,1cyclictest0-21swapper/223:41:152
15464992321,1cyclictest0-21swapper/122:02:261
15464992315,3cyclictest0-21swapper/122:28:331
1546999223,4cyclictest141rcu_preempt22:14:392
1546999220,22cyclictest0-21swapper/200:32:102
1546999220,21cyclictest0-21swapper/223:00:052
1546999220,18cyclictest0-21swapper/222:06:212
15457992221,1cyclictest0-21swapper/023:41:150
1547299210,20cyclictest0-21swapper/323:32:373
1546999210,21cyclictest0-21swapper/223:28:222
15464992117,3cyclictest0-21swapper/121:55:001
1545799215,3cyclictest0-21swapper/021:22:320
323512200,0sleep10-21swapper/122:33:371
1547299203,9cyclictest4608-21sh21:30:073
1547299200,20cyclictest0-21swapper/322:08:443
1546999200,20cyclictest0-21swapper/223:52:002
1546999200,20cyclictest0-21swapper/223:08:442
1546999200,20cyclictest0-21swapper/222:24:352
1546999200,18cyclictest0-21swapper/222:57:272
1546999200,16cyclictest0-21swapper/223:39:382
1546499207,9cyclictest0-21swapper/123:47:081
1546499207,10cyclictest13437-21processes20:10:231
1546499206,12cyclictest0-21swapper/123:19:501
15464992015,5cyclictest0-21swapper/122:09:571
15464992015,4cyclictest0-21swapper/100:20:081
15457992019,1cyclictest0-21swapper/023:35:400
1545799201,1cyclictest141rcu_preempt21:34:400
1547299198,10cyclictest7054-21unixbench-2d23:00:263
15469991913,6cyclictest0-21swapper/221:22:022
1546499198,9cyclictest0-21swapper/123:58:341
1546499196,12cyclictest0-21swapper/100:28:271
15464991916,2cyclictest0-21swapper/100:01:191
15464991916,1cyclictest8092-21diskmemload22:12:161
1546499190,18cyclictest0-21swapper/123:00:121
1545799197,7cyclictest12547-21diskstats23:50:160
15472991818,0cyclictest0-21swapper/321:42:173
1547299181,15cyclictest0-21swapper/322:32:043
1546999180,3cyclictest141rcu_preempt23:58:092
1546999180,18cyclictest0-21swapper/200:28:382
1546499187,9cyclictest0-21swapper/121:35:321
1546499187,10cyclictest0-21swapper/121:10:121
15464991814,3cyclictest0-21swapper/122:40:151
15464991814,3cyclictest0-21swapper/100:30:211
15464991812,5cyclictest0-21swapper/121:40:351
15464991811,5cyclictest0-21swapper/121:57:091
1545799180,18cyclictest0-21swapper/022:12:210
76482170,1sleep00-21swapper/023:00:530
15469991712,1cyclictest0-21swapper/221:11:502
1546499179,7cyclictest0-21swapper/121:21:061
1546499177,10cyclictest0-21swapper/123:51:181
15464991716,1cyclictest0-21swapper/123:20:331
15464991713,3cyclictest6539-21grep21:30:251
15464991713,3cyclictest0-21swapper/100:12:051
15464991711,5cyclictest0-21swapper/121:26:531
15457991717,0cyclictest0-21swapper/000:23:010
15457991716,1cyclictest0-21swapper/023:20:130
1545799170,17cyclictest0-21swapper/022:33:340
15469991615,1cyclictest0-21swapper/221:29:192
15469991615,1cyclictest0-21swapper/200:20:292
1546999160,1cyclictest0-21swapper/223:48:202
1546999160,15cyclictest0-21swapper/221:33:272
1546499165,10cyclictest0-21swapper/120:50:141
1546499164,7cyclictest0-21swapper/123:05:031
15464991613,2cyclictest8092-21diskmemload22:35:391
15464991610,5cyclictest13995-21wget00:15:141
15457991613,2cyclictest15480-21munin-plugin-st22:00:000
15457991610,6cyclictest0-21swapper/000:25:450
1547299158,7cyclictest8501-21/usr/sbin/munin21:10:213
1547299150,15cyclictest0-21swapper/321:37:303
1546999151,13cyclictest15926-21kworker/2:021:54:432
15469991510,2cyclictest32-21ksoftirqd/220:45:262
1546999150,14cyclictest0-21swapper/219:30:152
1546999150,12cyclictest0-21swapper/222:47:222
1546999150,12cyclictest0-21swapper/200:11:272
1546499159,5cyclictest6600-21fschecks_count21:05:171
1546499158,6cyclictest0-21swapper/120:25:301
1546499157,7cyclictest0-21swapper/123:13:151
1546499152,9cyclictest0-21swapper/119:30:151
15464991513,2cyclictest0-21swapper/121:45:211
15464991513,1cyclictest9432-21python00:35:261
15464991513,1cyclictest0-21swapper/120:45:151
15464991513,1cyclictest0-21swapper/120:10:001
15464991512,2cyclictest0-21swapper/121:15:141
15464991510,3cyclictest0-21swapper/119:55:261
1545799158,6cyclictest2344-21idleruntime-cro19:50:000
1547299142,1cyclictest141rcu_preempt23:15:573
15472991411,2cyclictest3748-21grep22:35:293
15472991410,3cyclictest20867-21grep23:55:293
1547299140,12cyclictest27478-21tr00:25:163
1547299140,12cyclictest0-21swapper/322:19:563
1547299140,0cyclictest0-21swapper/300:24:093
1546999141,3cyclictest16162-21kworker/2:000:15:572
1546999140,1cyclictest0-21swapper/223:15:142
15464991412,1cyclictest15447-21basename23:30:001
15464991412,1cyclictest0-21swapper/119:15:131
15464991411,2cyclictest0-21swapper/119:35:151
15464991410,3cyclictest0-21swapper/120:20:261
1545799146,7cyclictest18925-21munin-run23:10:000
15457991413,1cyclictest0-21swapper/021:48:450
1547299139,3cyclictest16147-21grep21:15:183
1547299138,2cyclictest3825-21ls19:50:223
1547299132,8cyclictest30116-21tr20:45:223
1547299132,10cyclictest2536-21ssh00:30:213
15472991313,0cyclictest0-21swapper/323:28:573
15472991312,1cyclictest0-21swapper/323:45:173
15472991312,1cyclictest0-21swapper/322:49:583
15472991310,2cyclictest19622-21sh22:25:013
1546999139,3cyclictest25493-21wc20:35:272
1546999131,2cyclictest0-21swapper/219:35:142
15469991311,1cyclictest0-21swapper/219:20:262
15469991310,2cyclictest6844-21cron00:10:002
1546999130,3cyclictest0-21swapper/221:46:392
1546499136,7cyclictest0-21swapper/122:50:191
15464991311,2cyclictest0-21swapper/122:18:211
1545799131,4cyclictest0-21swapper/022:19:340
15457991312,1cyclictest0-21swapper/000:14:340
15457991311,2cyclictest94-21kswapd021:40:030
1545799131,10cyclictest0-21swapper/023:27:120
1545799130,1cyclictest0-21swapper/022:06:210
1545799130,12cyclictest5737-21head23:45:170
1547299129,2cyclictest9607-21ssh21:55:193
1547299129,2cyclictest9562-21cat20:05:013
1547299129,2cyclictest9562-21cat20:05:013
1547299129,2cyclictest8428-21df00:35:173
1547299129,2cyclictest31763-21sed22:55:203
1547299129,2cyclictest31096-21grep21:25:223
1547299129,2cyclictest25522-21grep19:30:163
1547299129,2cyclictest25185-21idleruntime-cro19:30:003
1547299129,2cyclictest23182-21grep23:35:133
1547299129,2cyclictest21197-21grep19:20:173
1547299129,2cyclictest16835-21tr19:10:233
1547299129,2cyclictest1337-21cut19:45:203
1547299124,4cyclictest224-21systemd-journal20:59:593
15472991211,1cyclictest0-21swapper/322:26:403
15472991210,1cyclictest21300-21ssh22:03:393
1547299120,11cyclictest1462-21chrt23:20:133
1547299120,11cyclictest0-21swapper/321:46:353
1546999129,2cyclictest22869-21wc22:05:002
1546999129,2cyclictest16183-21grep20:15:292
1546999129,2cyclictest10537-21cut20:05:192
1546999128,3cyclictest844-21cut19:45:172
1546999128,3cyclictest3788-21sh23:20:482
1546999128,3cyclictest30886-21ssh22:32:122
1546999128,3cyclictest19706-21ssh23:32:142
1546999128,3cyclictest18757-21seq20:21:422
1546999122,9cyclictest7744-21snmpd21:15:362
1546999122,8cyclictest0-21swapper/219:59:592
1546999122,6cyclictest224-21systemd-journal20:00:142
1546999122,6cyclictest224-21systemd-journal20:00:132
1546999121,10cyclictest37850irq/131-snd_hda21:35:172
1546999120,3cyclictest0-21swapper/221:40:082
1546999120,1cyclictest0-21swapper/200:38:192
1546999120,12cyclictest0-21swapper/222:27:092
1546999120,11cyclictest4252-21sort21:00:182
1546999120,11cyclictest0-21swapper/219:15:142
1546499128,3cyclictest1891-21sed19:45:251
1546499126,5cyclictest24015-21/usr/sbin/munin20:35:161
1546499126,2cyclictest224-21systemd-journal00:08:181
1546499123,6cyclictest0-21swapper/119:25:021
15464991210,1cyclictest0-21swapper/120:40:151
1545799129,2cyclictest4348-31sh00:07:230
1545799129,2cyclictest4148-21df21:00:160
1545799129,2cyclictest31375-21tr19:40:230
1545799129,2cyclictest2100-21grep22:35:170
1545799129,2cyclictest18350-21tr19:15:150
1545799129,2cyclictest13050-21wc21:35:230
1545799128,3cyclictest18994-21ssh23:55:160
1545799123,2cyclictest0-21swapper/022:45:160
15457991211,1cyclictest0-21swapper/023:33:490
15457991211,1cyclictest0-21swapper/021:54:440
15457991210,2cyclictest0-21swapper/021:29:200
15457991210,1cyclictest945-21perf20:55:000
15457991210,1cyclictest23819-21grep19:25:190
15457991210,1cyclictest17817-21turbostat.cron19:15:000
15457991210,1cyclictest16767-21head22:00:180
1545799120,1cyclictest0-21swapper/000:15:480
1545799120,11cyclictest9882-21head22:40:170
1545799120,11cyclictest15012-21munin-plugin-st21:15:000
1547299118,2cyclictest7137-21awk20:00:003
1547299118,2cyclictest668-21taskset20:51:473
1547299118,2cyclictest5712-21ls21:05:003
1547299118,2cyclictest3919-21ssh21:50:503
1547299118,2cyclictest375-21sh00:05:023
1547299118,2cyclictest30990-21expr22:10:153
1547299118,2cyclictest28278-21taskset20:40:523
1547299118,2cyclictest26150-21ssh00:00:143
1547299118,2cyclictest25453-21sed22:50:293
1547299118,2cyclictest24060-21cut20:35:143
1547299118,2cyclictest23010-21grep21:20:163
1547299118,2cyclictest20982-21sed20:25:303
1547299118,2cyclictest19870-21expr23:10:163
1547299118,2cyclictest19020-21cron20:25:013
1547299118,2cyclictest17546-21ssh00:17:083
1547299118,2cyclictest15276-21sed20:15:193
1547299118,2cyclictest15203-21chrt23:06:303
1547299118,2cyclictest13156-21ntpq23:50:213
1547299118,2cyclictest12404-21cut20:10:163
1547299118,2cyclictest11176-21ssh22:40:343
1547299118,2cyclictest10000-21rm00:11:153
1547299117,3cyclictest6051-21grep21:05:153
1547299110,1cyclictest0-21swapper/323:43:063
1546999119,1cyclictest4279-21grep19:50:272
1546999119,1cyclictest32009-21cut20:50:182
1546999118,2cyclictest6112-21cut21:05:142
1546999118,2cyclictest27158-21if_err_enp0s31f00:00:212
1546999118,2cyclictest25058-21taskset19:28:312
1546999118,2cyclictest210472sleep220:25:372
1546999118,2cyclictest16930-21tr19:10:242
1546999118,2cyclictest11419-21ssh22:19:132
1546999110,1cyclictest0-21swapper/221:56:292
1546999110,10cyclictest0-21swapper/222:37:262
1546499118,2cyclictest3845-21cut21:00:161
1546499118,2cyclictest2647-21tr19:50:141
1546499118,2cyclictest23546-21taskset20:32:121
1546499118,2cyclictest15617-21cat20:15:221
1546499114,3cyclictest224-21systemd-journal19:15:001
15464991110,1cyclictest0-21swapper/119:20:261
1546499110,5cyclictest0-21swapper/119:40:171
1545799119,2cyclictest17074-21ssh21:15:260
1545799119,1cyclictest7070-21grep21:05:210
1545799119,1cyclictest32510-21ls00:05:010
1545799119,1cyclictest29659-21tail20:45:180
1545799119,1cyclictest26396-21cut20:40:010
1545799119,1cyclictest25562-21sed22:50:290
1545799119,1cyclictest2101-21grep00:30:180
1545799119,1cyclictest19821-21expr23:10:150
1545799119,1cyclictest18402-21expr20:20:280
1545799119,1cyclictest12591-21ssh22:20:030
1545799118,2cyclictest6556-21cat19:55:290
1545799118,2cyclictest2817-21grep20:55:280
1545799118,2cyclictest20393-21cat20:25:240
1545799118,2cyclictest10976-21ls20:05:240
1545799112,8cyclictest29820-21taskset19:37:530
1545799110,10cyclictest10197-21ssh00:35:540
1547299108,1cyclictest28248-21expr19:35:163
1547299107,2cyclictest22978-21gzip20:30:233
1547299107,2cyclictest19747-21sed19:15:273
1547299106,2cyclictest31424-21tr19:40:233
1546999108,2cyclictest13825-21awk20:10:272
1546999108,1cyclictest285642sleep220:44:322
1546999107,2cyclictest32125-21chrt19:41:452
1546999107,2cyclictest23564-21sh22:50:122
1546999107,2cyclictest22311-21grep20:30:172
1546999107,2cyclictest2080-21cat20:55:202
1546499107,2cyclictest0-21swapper/120:00:161
1546499107,2cyclictest0-21swapper/120:00:151
1546499101,3cyclictest0-21swapper/120:55:141
1545799109,1cyclictest27235-21taskset19:30:580
1545799108,1cyclictest7782-21cat20:00:170
1545799108,1cyclictest7782-21cat20:00:170
1545799108,1cyclictest2570-21chrt19:50:050
1545799108,1cyclictest21575-21grep19:20:210
1545799108,1cyclictest14609-21basename20:15:170
1545799108,1cyclictest12413-21grep20:10:160
1545799107,2cyclictest27475-21grep23:15:180
1545799106,1cyclictest500-21in:imuxsock20:34:550
1545799100,1cyclictest0-21swapper/020:40:150
154729997,1cyclictest9814-21cut20:05:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional