You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 03:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun May 31, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
724829682,9sleep00-21swapper/019:06:500
737029279,9sleep20-21swapper/219:08:242
723128663,6sleep30-21swapper/319:06:363
728028366,13sleep10-21swapper/119:07:161
256902330,0sleep20-21swapper/221:25:002
758599280,28cyclictest0-21swapper/221:42:492
757999280,0cyclictest0-21swapper/000:34:080
758599250,25cyclictest0-21swapper/222:18:252
758599240,24cyclictest0-21swapper/221:50:452
120732240,0sleep10-21swapper/121:15:521
758899231,22cyclictest0-21swapper/319:30:143
758599230,23cyclictest0-21swapper/223:01:172
757999230,1cyclictest0-21swapper/023:04:510
758899221,21cyclictest0-21swapper/321:50:593
7583992118,1cyclictest0-21swapper/122:45:001
758899200,2cyclictest0-21swapper/322:15:143
758899200,1cyclictest0-21swapper/323:50:193
7588991918,1cyclictest0-21swapper/322:25:263
758599180,1cyclictest141rcu_preempt22:05:152
758599180,16cyclictest0-21swapper/221:38:322
758399180,1cyclictest0-21swapper/123:27:321
757999187,5cyclictest25882-21perf20:55:000
757999183,14cyclictest1853-21apt-get20:05:130
757999180,2cyclictest0-21swapper/023:14:350
757999180,1cyclictest0-21swapper/022:33:070
7588991715,1cyclictest587-21diskmemload22:38:043
758899171,0cyclictest141rcu_preempt22:52:163
758599170,16cyclictest0-21swapper/223:17:222
7583991715,1cyclictest0-21swapper/100:30:131
758399171,3cyclictest0-21swapper/119:30:141
758399170,17cyclictest0-21swapper/123:30:151
757999170,0cyclictest0-21swapper/023:22:500
758899164,3cyclictest141rcu_preempt19:20:133
758899163,8cyclictest9441-21/usr/sbin/munin21:15:273
758899162,5cyclictest141rcu_preempt20:45:153
7588991611,2cyclictest39-21ksoftirqd/300:30:163
758899161,10cyclictest18336-21idleruntime-cro21:40:003
758899160,15cyclictest0-21swapper/323:20:143
758599160,16cyclictest0-21swapper/221:18:272
758599160,16cyclictest0-21swapper/200:02:082
758599160,14cyclictest0-21swapper/200:31:352
758599160,0cyclictest0-21swapper/221:27:142
7583991615,1cyclictest25-21ksoftirqd/100:00:321
757999160,1cyclictest0-21swapper/022:58:230
758899159,2cyclictest39-21ksoftirqd/321:59:443
7588991512,2cyclictest0-21swapper/321:20:233
758899150,2cyclictest0-21swapper/322:58:103
758899150,10cyclictest5069-21kworker/3:000:01:083
758599152,11cyclictest9446-21ssh00:08:352
7585991514,1cyclictest8228-21grep20:15:212
7585991513,2cyclictest0-21swapper/200:25:222
7585991511,3cyclictest1890-21taskset23:05:502
7583991514,1cyclictest0-21swapper/123:39:241
7583991514,1cyclictest0-21swapper/122:10:241
758399150,14cyclictest0-21swapper/122:51:481
758399150,0cyclictest0-21swapper/123:06:061
757999154,6cyclictest175842sleep022:36:440
757999152,7cyclictest19392-21sendmail-msp21:59:590
7579991514,1cyclictest0-21swapper/000:15:590
757999150,8cyclictest12397-21sendmail19:20:010
758899147,3cyclictest10207-21kworker/3:200:25:223
7588991413,1cyclictest381rcuc/323:34:203
758899141,1cyclictest141rcu_preempt21:33:323
7588991411,2cyclictest25863-21sh20:55:003
7588991411,2cyclictest10418-21wc20:20:193
7588991410,3cyclictest17000-21grep23:15:173
7588991410,3cyclictest16645-21grep20:35:153
7588991410,1cyclictest39-21ksoftirqd/321:11:523
758899140,4cyclictest0-21swapper/300:07:053
758899140,1cyclictest141rcu_preempt22:23:383
758899140,1cyclictest0-21swapper/300:21:583
758599144,9cyclictest6138-21ssh21:32:012
7585991410,2cyclictest28657-21grep21:00:152
758399145,6cyclictest38150irq/131-snd_hda22:00:471
7583991413,1cyclictest0-21swapper/121:45:261
758399141,12cyclictest0-21swapper/100:30:001
7583991410,3cyclictest8407-21ssh23:10:161
7583991410,3cyclictest38150irq/131-snd_hda21:20:211
7583991410,3cyclictest26545-21grep23:20:261
758399140,1cyclictest0-21swapper/123:58:091
757999141,5cyclictest6147-21basename22:50:010
7579991414,0cyclictest0-21swapper/021:46:330
7579991411,2cyclictest6352-21ssh00:06:220
7579991411,2cyclictest6322-21awk22:10:250
7579991410,3cyclictest17938-21grep21:20:140
758899139,3cyclictest39-21ksoftirqd/323:39:323
758899139,3cyclictest30553-21wc19:55:203
758899139,3cyclictest25324-21chrt19:45:153
758899135,7cyclictest0-21swapper/321:26:483
758899131,4cyclictest39-21ksoftirqd/319:40:203
758899131,2cyclictest587-21diskmemload23:49:063
7588991312,1cyclictest381rcuc/323:56:233
758899131,12cyclictest0-21swapper/322:31:043
7588991310,2cyclictest21498-21head00:15:183
7588991310,2cyclictest1412-21ssh23:25:173
758899130,12cyclictest22375-21cut22:40:143
758899130,12cyclictest0-21swapper/322:09:373
758599139,3cyclictest32-21ksoftirqd/200:25:002
7585991311,1cyclictest22275-21/usr/sbin/munin22:40:192
7585991310,2cyclictest5680-21grep22:30:152
7585991310,2cyclictest30249-21grep19:55:172
7585991310,2cyclictest27353-21tr19:50:142
7585991310,2cyclictest25932-21wc20:55:002
7585991310,2cyclictest17744-21grep19:30:152
7585991310,2cyclictest1309-21cat23:45:002
758399139,3cyclictest24088-21basename23:19:591
758399139,3cyclictest12796-21ssh00:10:161
758399136,4cyclictest15005-21/usr/sbin/munin22:55:171
758399133,10cyclictest25-21ksoftirqd/123:42:401
7583991312,1cyclictest0-21swapper/122:35:131
758399131,11cyclictest5706-21rm21:12:301
758399131,11cyclictest28797-21tr21:00:151
7583991310,2cyclictest3709-21python21:30:231
7583991310,2cyclictest16378-21sort20:35:001
758399130,12cyclictest14308-21wc22:15:231
757999139,3cyclictest30172-21grep21:00:270
757999139,3cyclictest30054-21ssh23:42:110
757999139,3cyclictest27193-21cut19:50:140
757999139,3cyclictest23874-21tail00:35:260
757999139,3cyclictest17348-21ssh23:15:210
757999136,3cyclictest11220-21ssh21:15:290
757999131,7cyclictest26672-21cron21:45:000
757999131,10cyclictest29075-21apt-get00:20:120
7579991310,2cyclictest9648-21cron23:50:000
7579991310,2cyclictest26413-21sed19:45:260
757999130,11cyclictest0-21swapper/023:50:150
757999130,0cyclictest0-21swapper/023:05:490
758899129,3cyclictest0-21swapper/321:42:353
758899129,2cyclictest39-21ksoftirqd/323:02:593
758899129,2cyclictest39-21ksoftirqd/320:00:213
758899129,2cyclictest31836-21cut23:05:143
758899129,2cyclictest14724-21grep20:30:133
758899129,2cyclictest14039-21cat20:30:003
758899129,2cyclictest13370-21chrt23:13:123
758899123,2cyclictest141rcu_preempt22:01:273
758899122,10cyclictest39-21ksoftirqd/300:38:213
758599129,2cyclictest7399-21grep22:50:152
758599129,2cyclictest31501-21sed22:25:262
758599129,2cyclictest21797-21ssh22:20:152
758599129,2cyclictest1778-21grep21:10:172
758599129,2cyclictest11784-21cut21:55:112
758599128,3cyclictest26646-21ssh23:20:272
758599128,3cyclictest25014-21tr19:45:152
758599128,3cyclictest18885-21wc20:40:012
758599128,3cyclictest14465-21sh22:35:142
7585991210,2cyclictest291irq_work/223:55:182
758399129,2cyclictest7247-21tr20:15:141
758399129,2cyclictest2938-21sh22:47:321
758399129,2cyclictest2858-21uname23:45:191
758399129,2cyclictest28285-21wc21:00:011
758399129,2cyclictest27721-21grep19:50:161
758399129,2cyclictest26676-21cron21:45:001
758399129,2cyclictest1601-21awk20:05:001
758399129,2cyclictest12974-21cut20:25:211
758399128,3cyclictest587-21diskmemload23:50:261
758399128,3cyclictest3947-21cron20:09:591
758399128,3cyclictest38150irq/131-snd_hda22:32:141
758399128,3cyclictest31507-21grep22:25:261
758399128,2cyclictest16157-21tr19:25:191
758399121,2cyclictest0-21swapper/119:50:001
757999129,2cyclictest9974-21basename21:34:590
757999129,2cyclictest31701-21ssh00:02:320
757999129,2cyclictest26510-21tr21:25:160
757999129,2cyclictest20220-21cut19:35:130
757999129,2cyclictest17334-21grep20:35:180
757999129,2cyclictest14767-21perf19:25:000
757999129,2cyclictest14293-21basename20:30:110
757999129,2cyclictest13601-21grep22:15:160
757999129,2cyclictest13546-21head20:25:270
757999129,2cyclictest13014-21sh00:30:000
757999129,2cyclictest12736-21grep00:10:160
757999128,2cyclictest8520-21head19:10:150
757999127,5cyclictest11483-21apache222:29:260
757999127,3cyclictest16691-21tail19:25:250
757999124,6cyclictest602-21ssh23:25:140
757999120,2cyclictest0-21swapper/019:40:120
757999120,0cyclictest0-21swapper/023:38:170
758899118,2cyclictest5872-21ssh22:10:213
758899118,2cyclictest30635-21ls21:05:003
758899118,2cyclictest28228-21cron21:00:013
758899118,2cyclictest20500-21sed19:35:143
758899118,2cyclictest12400-21basename19:20:013
758899112,8cyclictest12311-21/usr/sbin/munin00:10:153
758899111,8cyclictest0-21swapper/322:46:113
758899110,11cyclictest0-21swapper/321:45:563
758599119,1cyclictest23593-21wc20:50:012
758599118,3cyclictest32-21ksoftirqd/200:35:372
758599118,2cyclictest9645-21taskset19:11:362
758599118,2cyclictest9484-21ssh23:30:132
758599118,2cyclictest7335-21ssh22:11:092
758599118,2cyclictest4245-21sh23:45:472
758599118,2cyclictest32265-21ssh22:45:242
758599118,2cyclictest28473-21cut21:45:202
758599118,2cyclictest23176-21ssh00:15:492
758599118,2cyclictest20751-21expr22:00:182
758599118,2cyclictest1877-21ssh23:25:222
758599118,2cyclictest17526-21ssh22:55:492
758599118,2cyclictest17222-21ssh23:35:062
758599118,2cyclictest15295-21grep19:25:142
758599118,2cyclictest11295-21latency23:50:212
758599118,2cyclictest11125-21ssh23:11:302
758599115,2cyclictest225-21systemd-journal20:10:002
7585991111,0cyclictest0-21swapper/200:10:142
758399118,2cyclictest9196-21ssh21:53:381
758399118,2cyclictest8579-21grep19:10:151
758399118,2cyclictest6367-21ssh00:06:231
758399118,2cyclictest6134-21sed20:10:251
758399118,2cyclictest31296-21taskset19:56:141
758399118,2cyclictest29366-21ssh22:05:181
758399118,2cyclictest2910-21sh00:23:431
758399118,2cyclictest28731-21ssh23:03:031
758399118,2cyclictest28478-21ssh21:25:401
758399118,2cyclictest28041-21ssh22:24:121
758399118,2cyclictest24625-21sh00:35:581
758399118,2cyclictest23591-21ssh00:16:091
758399118,2cyclictest21341-21taskset20:45:091
758399118,2cyclictest11559-21grep19:15:211
758399118,2cyclictest11248-21sh21:55:041
758399118,2cyclictest10777-21cut20:20:231
758399117,3cyclictest18324-21munin-run21:40:001
7583991111,0cyclictest0-21swapper/120:50:241
757999118,2cyclictest9499-21cut20:20:130
757999118,2cyclictest7764-21ssh22:50:180
757999118,2cyclictest7253-21tr20:15:140
757999118,2cyclictest3842-21cat22:10:010
757999118,2cyclictest2976-21ssh21:50:080
757999118,2cyclictest29597-21wget19:55:140
757999118,2cyclictest28543-21ssh22:24:350
757999118,2cyclictest22499-21tr20:45:210
757999118,2cyclictest20230-21grep20:40:220
757999118,2cyclictest20230-21grep20:40:210
757999118,2cyclictest16062-21ssh23:34:210
757999118,2cyclictest14258-21ssh21:36:480
757999111,9cyclictest0-21swapper/021:10:370
7579991110,1cyclictest0-21swapper/023:55:110
757999110,5cyclictest20453-21sh22:00:150
757999110,10cyclictest0-21swapper/022:41:150
758899108,1cyclictest7503-21expr20:15:143
758899108,1cyclictest32659-21taskset21:05:423
758899108,1cyclictest2971-21ntpq20:05:213
758899108,1cyclictest27349-21expr19:50:143
758899108,1cyclictest25928-21cut23:40:133
758899108,1cyclictest15010-21basename19:25:113
758899107,2cyclictest5695-21wc20:10:213
758899107,2cyclictest19710-21head20:40:173
758899107,2cyclictest19710-21head20:40:163
758599108,1cyclictest9791-21expr20:20:152
758599108,1cyclictest6208-21chrt20:10:472
758599108,1cyclictest241242sleep219:40:342
758599108,1cyclictest19202-21grep20:40:142
758599108,1cyclictest19202-21grep20:40:142
758599108,1cyclictest16038-21taskset20:31:142
758599108,1cyclictest12072-21taskset19:16:312
758599107,2cyclictest473-21taskset21:09:112
758599107,2cyclictest21842-21taskset19:37:002
758599107,2cyclictest12575-21grep20:25:182
758599106,1cyclictest13726-21cat19:20:182
758599104,5cyclictest26706-21/usr/sbin/munin20:55:172
758599100,9cyclictest31856-21taskset20:00:092
758399108,1cyclictest13905-21grep19:20:201
758399107,2cyclictest31116-21grep21:05:131
758399107,2cyclictest23598-21tr19:40:221
758399107,2cyclictest20718-21expr19:35:151
758399107,2cyclictest20141-21ntpq20:40:201
758399107,2cyclictest20141-21ntpq20:40:201
758399107,2cyclictest16960-21expr20:35:161
757999108,1cyclictest6249-21taskset20:11:200
757999107,2cyclictest31678-21sed21:05:160
757999107,2cyclictest1801-21date20:05:010
757999107,2cyclictest17601-21apache_accesses19:30:120
75889997,1cyclictest8596-21tail19:10:163
75799997,1cyclictest260912sleep020:55:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional