You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 17:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 20, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2328229776,16sleep00-21swapper/007:08:510
2317529066,9sleep20-21swapper/207:07:292
2336428471,9sleep30-21swapper/307:09:543
2302427564,6sleep10-21swapper/107:05:311
2345199260,25cyclictest0-21swapper/109:46:501
23450992524,1cyclictest0-21swapper/009:35:360
2345199241,22cyclictest0-21swapper/110:55:131
2345399230,19cyclictest0-21swapper/310:05:133
2345199230,22cyclictest0-21swapper/110:40:111
2345199220,3cyclictest0-21swapper/112:00:181
23453992018,1cyclictest475-21in:imuxsock12:18:363
2345299202,4cyclictest0-21swapper/210:10:132
23452992014,1cyclictest0-21swapper/212:25:422
2345299201,1cyclictest141rcu_preempt09:11:192
2345299201,1cyclictest141rcu_preempt09:11:182
23451992020,0cyclictest0-21swapper/109:15:581
2345199200,2cyclictest0-21swapper/111:41:261
23453991914,4cyclictest0-21swapper/312:01:153
2345299190,18cyclictest0-21swapper/210:58:292
23450991917,1cyclictest475-21in:imuxsock10:53:460
126292190,0sleep00-21swapper/011:40:140
2345399184,12cyclictest8525-21grep11:00:003
2345399181,4cyclictest141rcu_preempt09:36:353
2345299180,17cyclictest0-21swapper/211:45:462
2345199180,17cyclictest0-21swapper/109:58:481
23450991817,1cyclictest0-21swapper/012:03:520
2345099180,1cyclictest0-21swapper/011:34:300
2345099180,14cyclictest0-21swapper/011:59:200
2345399177,6cyclictest537-21snmpd09:49:593
23453991714,2cyclictest39-21ksoftirqd/311:07:313
23452991715,1cyclictest475-21in:imuxsock11:42:352
2345299170,17cyclictest0-21swapper/210:01:392
23451991715,1cyclictest0-21swapper/110:13:481
2345399162,2cyclictest141rcu_preempt10:01:473
2345399161,12cyclictest0-21swapper/312:34:413
2345299163,1cyclictest141rcu_preempt10:05:152
23452991615,1cyclictest0-21swapper/209:55:122
23452991614,1cyclictest32-21ksoftirqd/210:21:572
23452991613,2cyclictest55650irq/127-enp2s0-09:28:062
2345299160,16cyclictest0-21swapper/209:53:232
2345299160,0cyclictest141rcu_preempt12:14:182
2345199160,1cyclictest16469-21diskmemload12:36:451
2345199160,15cyclictest0-21swapper/109:12:091
2345199160,15cyclictest0-21swapper/109:12:081
2345099160,15cyclictest0-21swapper/009:59:310
163032160,0sleep20-21swapper/212:20:192
23453991511,3cyclictest23645-21head08:15:173
2345399150,2cyclictest0-21swapper/311:55:203
2345399150,0cyclictest0-21swapper/311:37:463
2345299151,0cyclictest141rcu_preempt11:51:532
2345299151,0cyclictest141rcu_preempt11:23:482
2345299150,1cyclictest141rcu_preempt11:37:522
2345199154,10cyclictest0-21swapper/112:13:191
2345199150,0cyclictest0-21swapper/111:34:301
23450991512,2cyclictest12838-21cat09:45:170
2345099151,1cyclictest141rcu_preempt11:35:210
316162140,0sleep00-21swapper/012:10:190
23453991414,0cyclictest0-21swapper/309:44:003
23453991410,3cyclictest17920-21cut09:10:173
23453991410,3cyclictest17920-21cut09:10:163
23452991412,1cyclictest32-21ksoftirqd/210:36:322
23452991411,2cyclictest1172-21grep08:35:192
2345299140,13cyclictest0-21swapper/212:38:422
2345299140,13cyclictest0-21swapper/211:57:022
2345299140,0cyclictest141rcu_preempt10:25:362
2345199147,5cyclictest38150irq/131-snd_hda10:49:561
2345199146,4cyclictest225-21systemd-journal09:05:011
23451991410,3cyclictest2172-21cron08:39:591
2345199140,11cyclictest0-21swapper/110:09:341
23450991414,0cyclictest0-21swapper/010:39:590
23450991413,1cyclictest0-21swapper/007:30:130
2345099141,2cyclictest141rcu_preempt11:08:580
23450991411,2cyclictest18784-21cut08:05:160
2345399139,3cyclictest25839-21grep10:50:203
2345399133,4cyclictest0-21swapper/307:55:133
23453991311,1cyclictest2949-21ssh10:17:523
23453991310,2cyclictest26023-21grep08:20:163
23453991310,2cyclictest25464-21grep12:25:253
23453991310,2cyclictest16799-21sort10:45:153
23453991310,2cyclictest15861-21grep12:20:163
23453991310,2cyclictest13033-21ssh11:02:003
23453991310,2cyclictest1277-21tr09:20:013
23453991310,2cyclictest11500-21ntpq09:25:203
23453991310,2cyclictest10765-21awk07:50:003
23453991310,1cyclictest0-21swapper/310:39:153
2345399130,12cyclictest2562-21expr11:15:163
2345299139,3cyclictest2984-21grep07:30:252
2345299138,5cyclictest32-21ksoftirqd/208:30:132
2345299138,3cyclictest32-21ksoftirqd/209:40:012
2345299132,11cyclictest32-21ksoftirqd/209:45:122
2345299132,11cyclictest26787-21cups-browsed07:40:222
23452991311,1cyclictest32-21ksoftirqd/212:32:502
23452991311,1cyclictest32-21ksoftirqd/209:19:022
23452991310,2cyclictest8638-21ssh11:00:012
23452991310,2cyclictest32-21ksoftirqd/211:31:072
2345299130,9cyclictest0-21swapper/212:01:022
2345299130,13cyclictest0-21swapper/211:25:142
2345299130,12cyclictest6098-21grep08:45:212
2345199138,4cyclictest6993-21sshd12:34:351
23451991310,2cyclictest6511-21tail08:45:241
23451991310,2cyclictest475-21in:imuxsock11:26:361
23451991310,2cyclictest4001-21grep11:15:281
23451991310,2cyclictest22172-21tr12:05:111
23451991310,2cyclictest10334-21grep08:55:161
23451991310,2cyclictest10235-21tr11:00:191
2345199130,2cyclictest0-21swapper/110:54:221
2345199130,12cyclictest15883-21cut08:00:131
2345099139,3cyclictest19143-21grep09:10:270
2345099139,3cyclictest19143-21grep09:10:270
2345099131,2cyclictest141rcu_preempt11:24:350
23450991312,1cyclictest0-21swapper/010:10:130
23450991311,2cyclictest0-21swapper/009:41:090
2345399129,2cyclictest24162-21tr07:10:133
2345399129,2cyclictest20302-21uname11:25:203
2345399129,2cyclictest16456-21cron09:10:003
2345399129,2cyclictest15499-21head10:25:163
2345399129,2cyclictest1445-21chrt11:52:423
2345399129,2cyclictest1032-21basename07:30:013
2345399128,3cyclictest3141-21ls09:20:203
2345399128,2cyclictest7760-21grep10:40:123
2345399127,4cyclictest22336-21sed11:45:253
2345299129,2cyclictest7934-21grep08:50:172
2345299129,2cyclictest4037-21tail09:40:182
2345299129,2cyclictest30904-21cut10:15:162
2345299129,2cyclictest30091-21tr07:20:182
2345299129,2cyclictest27850-21cut08:25:142
2345299129,2cyclictest25350-21cut10:50:162
2345299129,2cyclictest2430-21tr11:15:162
2345299129,2cyclictest20754-21tr08:10:132
2345299129,2cyclictest1553-21apt-config11:15:002
2345299129,2cyclictest15050-21cut09:05:172
2345299129,2cyclictest13625-21grep09:00:262
2345299124,6cyclictest32-21ksoftirqd/210:40:122
2345299120,11cyclictest23042-21wget08:15:142
2345199129,2cyclictest7400-21head12:15:151
2345199129,2cyclictest1883-21cut07:30:151
2345199129,2cyclictest16610-21ssh12:20:211
2345199129,2cyclictest1534-21sort11:15:001
2345199128,3cyclictest5192-21head09:40:271
2345199128,3cyclictest20086-21ssh09:50:031
2345199127,4cyclictest31250-21ssh11:50:421
2345199127,4cyclictest13815-21ssh10:24:421
2345199120,2cyclictest0-21swapper/111:48:351
2345199120,0cyclictest0-21swapper/110:28:351
2345099129,2cyclictest8368-21cron07:45:010
2345099129,2cyclictest15510-21turbostat08:00:000
2345099129,2cyclictest10291-21grep07:45:260
2345099125,4cyclictest19801-21/usr/sbin/munin09:30:200
2345099120,2cyclictest0-21swapper/012:34:510
2345099120,11cyclictest779-21grep08:35:160
2345399118,2cyclictest6748-21df07:40:143
2345399118,2cyclictest6727-21cat10:20:143
2345399118,2cyclictest31860-21ssh09:56:143
2345399118,2cyclictest27039-21cat11:10:203
2345399118,2cyclictest25667-21ssh10:11:573
2345399118,2cyclictest23425-21ssh12:05:193
2345399118,2cyclictest21120-21head09:50:163
2345399118,2cyclictest20899-21expr08:10:143
2345399118,2cyclictest19402-21ssh09:30:163
2345399118,2cyclictest1901-21ssh12:11:553
2345399118,2cyclictest16541-21grep08:00:173
2345399118,2cyclictest16143-21ssh11:42:013
2345399118,2cyclictest13109-21cron07:55:003
2345399118,2cyclictest12234-21chrt12:37:063
2345399118,2cyclictest12001-21head11:20:223
2345399115,5cyclictest29186-21sensors11:30:243
2345399114,3cyclictest225-21systemd-journal07:25:003
2345399111,7cyclictest0-21swapper/310:30:133
2345399110,9cyclictest0-21swapper/308:40:113
2345299119,1cyclictest311rcuc/212:05:212
2345299118,2cyclictest9800-21ntpq07:45:212
2345299118,2cyclictest4608-21tail07:35:172
2345299118,2cyclictest28026-21head07:15:252
2345299118,2cyclictest2656-21awk08:40:132
2345299118,2cyclictest25824-21sh10:30:562
2345299118,2cyclictest2291-21ssh09:20:142
2345299118,2cyclictest21235-21ssh10:47:502
2345299118,2cyclictest19942-21chrt08:05:572
2345299118,2cyclictest18920-21wc09:30:142
2345299118,2cyclictest11209-21sed08:55:262
2345299118,2cyclictest10655-21sh12:17:002
2345299116,1cyclictest141rcu_preempt08:24:592
2345299110,10cyclictest19085-21wc11:05:242
2345199119,1cyclictest3181-21head08:40:161
2345199119,1cyclictest25599-21grep07:10:251
2345199119,1cyclictest0-21swapper/109:35:231
2345199118,2cyclictest5938-21tail11:55:161
2345199118,2cyclictest4340-21ssh10:18:571
2345199118,2cyclictest3906-21cut11:35:151
2345199118,2cyclictest32141-21expr10:35:141
2345199118,2cyclictest26505-21ssh10:31:271
2345199118,2cyclictest26389-21cut07:15:131
2345199118,2cyclictest25810-21ssh12:25:401
2345199118,2cyclictest20718-21grep08:10:131
2345199118,2cyclictest15117-21sed09:05:171
2345199118,2cyclictest12920-21ssh11:20:401
2345199118,2cyclictest11223-21seq10:03:481
2345199117,3cyclictest30717-21taskset07:20:401
2345199112,7cyclictest18168-21munin-run09:30:001
23451991110,1cyclictest0-21swapper/108:15:151
2345199110,10cyclictest30664-21df08:30:161
2345199110,10cyclictest21978-21ssh11:07:271
2345199110,10cyclictest19024-21expr09:30:141
2345099119,1cyclictest258512sleep010:30:570
2345099119,1cyclictest18053-21ssh10:07:350
2345099119,1cyclictest12741-21grep09:00:170
2345099118,3cyclictest26751-21apache211:49:320
2345099118,2cyclictest8443-21grep08:50:220
2345099118,2cyclictest7467-21rm10:40:050
2345099118,2cyclictest6839-21ssh10:20:140
2345099118,2cyclictest6371-21gzip10:00:220
2345099118,2cyclictest5427-21rm09:21:420
2345099118,2cyclictest4683-21cat08:45:010
2345099118,2cyclictest4477-21ssh10:56:550
2345099118,2cyclictest32125-21grep10:15:270
2345099118,2cyclictest30511-21sensors_fan11:50:240
2345099118,2cyclictest29571-21cut07:20:140
2345099118,2cyclictest26008-21cron07:15:000
2345099118,2cyclictest23514-21ssh12:25:120
2345099118,2cyclictest21274-21grep08:10:150
2345099118,2cyclictest21201-21ssh12:23:350
2345099118,2cyclictest16080-21grep10:25:200
2345099118,2cyclictest14929-21df09:05:160
2345099118,2cyclictest13277-21ssh11:02:100
2345099118,2cyclictest11195-21grep08:55:260
2345099118,2cyclictest10499-21tr09:25:130
2345099117,1cyclictest0-21swapper/012:16:330
2345099110,10cyclictest27804-21sed11:10:250
2345399108,1cyclictest7513-21seq08:50:143
2345399108,1cyclictest464-21expr08:35:153
2345399108,1cyclictest30470-21ls08:30:143
2345399108,1cyclictest296512sleep308:27:353
2345399108,1cyclictest19033-21cat08:05:173
2345399108,1cyclictest13976-21taskset09:03:423
2345399107,2cyclictest27100-21cut07:15:153
2345399107,2cyclictest1538-21tr07:30:133
2345399102,4cyclictest225-21systemd-journal07:37:363
2345399100,9cyclictest6428-21cat08:45:253
2345299108,1cyclictest31879-21sed07:25:152
2345299108,1cyclictest145092sleep207:55:192
2345299108,1cyclictest11406-21cat07:50:132
2345299107,2cyclictest24635-21grep07:10:162
2345299107,2cyclictest16027-21tr08:00:152
2345199108,1cyclictest7317-21wc07:40:191
2345199108,1cyclictest3892-21aten2_r7power_c07:35:121
2345199108,1cyclictest19298-21awk08:05:201
2345199108,1cyclictest11065-21aten2_r7power_e07:50:121
2345199107,2cyclictest86412sleep107:45:131
2345199107,2cyclictest7671-21expr08:50:161
2345199107,2cyclictest28185-21expr08:25:131
2345199107,2cyclictest1599-21sh09:20:061
2345199106,3cyclictest25533-21tr08:20:131
2345099108,2cyclictest5237-21expr08:45:130
2345099108,2cyclictest5225-21which07:35:210
2345099108,2cyclictest3448-21ntp_states11:15:230
2345099108,1cyclictest28971-21sh09:16:230
2345099108,1cyclictest20432-21ssh09:50:130
2345099108,1cyclictest12365-21ssh12:37:130
2345099108,1cyclictest11077-21wget07:50:120
2345099107,2cyclictest31800-21grep08:30:260
2345099107,2cyclictest26618-21expr07:15:140
2345099107,2cyclictest24533-21grep08:15:260
2345099107,2cyclictest22717-21ssh12:05:140
2345099107,2cyclictest20991-21grep11:25:270
2345099100,9cyclictest15993-21ssh10:45:090
234539994,3cyclictest9592-21apt-get08:55:143
234519997,1cyclictest32465-21ntpq07:25:201
234519997,1cyclictest13643-21grep07:55:141
234509998,1cyclictest266972sleep008:20:230
234509998,1cyclictest16107-21expr08:00:140
234509997,2cyclictest31635-21cat07:25:140
234509987,1cyclictest27830-21hostname08:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional