You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 14:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 16, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
706029783,10sleep20-21swapper/207:06:552
728529166,9sleep00-21swapper/007:09:510
696428861,9sleep30-21swapper/307:05:393
709428758,6sleep10-21swapper/107:07:221
739199270,26cyclictest0-21swapper/211:24:102
739199260,25cyclictest0-21swapper/210:18:102
7388992623,2cyclictest809-21diskmemload10:26:011
738899262,24cyclictest14866-21apache211:29:031
738899254,2cyclictest25-21ksoftirqd/111:59:021
738899253,3cyclictest0-21swapper/110:40:321
738899253,0cyclictest0-21swapper/109:18:211
7388992521,3cyclictest0-21swapper/109:48:221
738599250,23cyclictest0-21swapper/012:25:110
7388992421,2cyclictest809-21diskmemload11:01:481
7388992416,7cyclictest25-21ksoftirqd/109:13:281
738899241,2cyclictest0-21swapper/110:21:281
739499233,4cyclictest141rcu_preempt11:08:563
739199230,1cyclictest0-21swapper/211:01:472
738899239,13cyclictest0-21swapper/110:52:351
738899230,1cyclictest0-21swapper/109:36:171
739499220,21cyclictest0-21swapper/312:32:583
739499220,1cyclictest528-21kworker/3:011:35:233
7391992222,0cyclictest0-21swapper/211:46:132
739199221,18cyclictest0-21swapper/212:25:112
739199220,21cyclictest0-21swapper/212:04:322
739199220,20cyclictest0-21swapper/212:05:432
7388992218,3cyclictest809-21diskmemload10:02:341
7388992216,5cyclictest809-21diskmemload11:05:361
7388992216,5cyclictest0-21swapper/111:36:041
739199210,11cyclictest0-21swapper/210:26:282
7388992116,2cyclictest0-21swapper/111:40:211
7388992112,7cyclictest0-21swapper/109:52:371
738899210,1cyclictest0-21swapper/110:35:051
255572210,0sleep00-21swapper/010:02:360
739499200,1cyclictest0-21swapper/311:30:133
7391992019,1cyclictest0-21swapper/210:11:392
739199200,20cyclictest0-21swapper/211:10:472
7388992017,2cyclictest809-21diskmemload09:34:071
7388992012,7cyclictest0-21swapper/111:15:141
738599206,8cyclictest31667-21lockfile-create08:00:010
7385992018,1cyclictest0-21swapper/012:14:060
738599200,20cyclictest0-21swapper/012:07:450
7394991919,0cyclictest0-21swapper/310:26:133
739199190,19cyclictest0-21swapper/212:18:372
7388991918,1cyclictest0-21swapper/112:33:441
7388991916,1cyclictest0-21swapper/112:38:051
7388991914,5cyclictest0-21swapper/110:14:351
738899190,19cyclictest0-21swapper/110:56:211
738599191,2cyclictest0-21swapper/009:49:590
738599190,16cyclictest1927-21ssh11:25:140
739499180,14cyclictest0-21swapper/312:10:133
7391991817,1cyclictest0-21swapper/209:33:072
739199180,17cyclictest0-21swapper/209:51:242
738899184,13cyclictest0-21swapper/109:56:021
7388991815,3cyclictest0-21swapper/112:08:181
7388991810,8cyclictest809-21diskmemload12:20:501
738599186,6cyclictest13953-21perf08:30:000
7385991818,0cyclictest0-21swapper/011:38:110
738899175,12cyclictest0-21swapper/112:04:101
7388991717,0cyclictest0-21swapper/108:15:231
7388991715,1cyclictest18148-21cron11:35:001
7388991713,4cyclictest0-21swapper/111:47:021
7388991713,4cyclictest0-21swapper/110:07:211
738899171,11cyclictest17419-21runrttasks10:16:471
738899170,6cyclictest241rcuc/110:30:481
738599173,9cyclictest3800-21apt12:05:010
738599170,16cyclictest0-21swapper/009:40:560
7394991614,1cyclictest7379-21cyclictest09:52:453
7394991613,3cyclictest55650irq/127-enp2s0-10:12:383
7394991613,1cyclictest7379-21cyclictest09:40:013
739499160,1cyclictest0-21swapper/311:59:153
739499160,1cyclictest0-21swapper/309:56:213
739199160,16cyclictest0-21swapper/209:21:182
739199160,13cyclictest0-21swapper/209:10:132
738899166,6cyclictest0-21swapper/112:15:161
738899162,12cyclictest0-21swapper/108:10:141
7388991613,2cyclictest0-21swapper/108:00:181
7388991611,4cyclictest25965-21ls07:45:221
7388991610,5cyclictest0-21swapper/112:25:171
7388991610,4cyclictest0-21swapper/111:11:161
7385991614,2cyclictest13-21ksoftirqd/011:55:010
7385991614,1cyclictest25484-21sed11:20:010
739499156,5cyclictest225-21systemd-journal09:00:003
739499153,9cyclictest0-21swapper/310:50:073
7394991511,3cyclictest23644-21grep10:40:133
7391991512,2cyclictest26621-21apache_volume09:25:122
739199150,1cyclictest0-21swapper/211:43:042
739199150,1cyclictest0-21swapper/210:21:452
739199150,13cyclictest0-21swapper/209:38:422
7388991513,1cyclictest1737-21ssh10:45:361
738599157,7cyclictest2309-21/usr/sbin/munin11:05:240
738599151,8cyclictest9175-21munin-run08:20:010
7385991514,1cyclictest0-21swapper/010:17:310
739499146,7cyclictest7323-21cat10:50:003
739499146,7cyclictest27877-21/usr/sbin/munin09:45:193
7394991410,3cyclictest20893-21grep08:40:273
739499140,3cyclictest0-21swapper/309:39:373
739199147,2cyclictest21616-21head10:00:172
7391991414,0cyclictest0-21swapper/211:19:422
7391991410,3cyclictest8136-21grep10:50:152
7391991410,2cyclictest18352-21wc11:35:012
739199140,14cyclictest0-21swapper/212:11:072
739199140,14cyclictest0-21swapper/211:53:102
738899149,4cyclictest16365-21idleruntime-cro08:35:011
738899146,6cyclictest0-21swapper/108:49:591
738899142,6cyclictest1075-21runrttasks11:24:481
7388991412,1cyclictest12708-21wc11:50:181
738899140,13cyclictest28018-21sed09:25:211
738599145,8cyclictest13124-21munin-run10:15:000
7385991411,2cyclictest20947-21grep08:40:270
7385991410,3cyclictest30337-21cat10:05:200
7385991410,2cyclictest28616-21wc11:40:230
7385991410,2cyclictest27967-21grep08:55:240
739499139,3cyclictest9354-21grep11:10:153
739499139,3cyclictest23676-21wc12:35:213
739499132,1cyclictest0-21swapper/310:32:563
7394991310,2cyclictest4442-21ssh11:25:513
7394991310,2cyclictest4169-21wc09:30:213
7394991310,2cyclictest23287-21grep07:40:213
7394991310,2cyclictest21096-21tr10:00:153
7394991310,2cyclictest18731-21tr11:15:173
7394991310,2cyclictest15867-21grep08:30:243
7394991310,2cyclictest14525-21sleep07:25:003
739499130,1cyclictest0-21swapper/312:20:103
739499130,12cyclictest0-21swapper/311:24:223
739199133,6cyclictest225-21systemd-journal10:50:002
7391991311,1cyclictest0-21swapper/210:57:582
7391991310,2cyclictest7869-21grep07:10:152
7391991310,2cyclictest30236-21grep07:55:172
7391991310,2cyclictest20303-21grep09:40:162
7391991310,2cyclictest14100-21ssh12:30:132
739199130,13cyclictest0-21swapper/212:36:322
738899138,4cyclictest27000-21ssh09:44:401
7388991312,1cyclictest241rcuc/108:53:581
7388991312,1cyclictest0-21swapper/109:21:461
7388991310,2cyclictest31227-21grep09:05:131
7388991310,2cyclictest16945-21grep08:35:121
738599139,2cyclictest5451-21grep08:10:180
738599137,2cyclictest20291-21grep11:55:130
738599132,6cyclictest11867-21basename09:55:010
738599131,6cyclictest23720-21munin-run08:49:590
7385991310,2cyclictest30017-21basename10:25:000
7385991310,2cyclictest25535-21sed07:45:180
7385991310,2cyclictest1439-21wc11:25:000
7385991310,2cyclictest11226-21grep07:15:210
738599130,1cyclictest0-21swapper/010:25:110
739499129,2cyclictest8329-21grep07:10:173
739499129,2cyclictest7756-21head08:15:163
739499129,2cyclictest1838-21basename08:05:003
739499129,2cyclictest12371-21grep11:50:163
739499128,3cyclictest15676-21cut10:35:153
739499128,2cyclictest4572-21tr12:05:143
739499127,3cyclictest2753-21cat11:45:003
739499125,5cyclictest809-21diskmemload09:26:233
7394991211,1cyclictest0-21swapper/310:19:163
739199129,2cyclictest20521-21sed11:35:262
739199129,2cyclictest15744-21grep07:25:172
739199129,2cyclictest14376-21cron10:34:592
739199128,3cyclictest2850-21grep08:05:172
739199128,3cyclictest2850-21grep08:05:172
739199128,3cyclictest22844-21systemctl10:40:002
739199128,2cyclictest991-21cat09:09:592
739199126,2cyclictest449-21irqbalance11:06:012
739199125,3cyclictest225-21systemd-journal08:30:012
739199120,11cyclictest0-21swapper/210:05:122
738899129,3cyclictest25-21ksoftirqd/108:06:581
738899129,3cyclictest25-21ksoftirqd/108:06:581
738899129,2cyclictest8874-21wc07:10:221
738899129,2cyclictest22833-21grep07:40:181
738899129,2cyclictest15530-21df07:25:161
738899129,2cyclictest14616-21head12:10:271
738899126,5cyclictest12086-21cpuspeed_turbos08:25:151
738599129,2cyclictest30580-21sh12:20:190
738599129,2cyclictest26124-21cron09:24:590
738599128,3cyclictest32232-21uname09:05:200
738599120,2cyclictest0-21swapper/007:40:140
738599120,1cyclictest0-21swapper/010:51:250
738599120,11cyclictest12183-21grep11:30:250
739499119,1cyclictest30377-21head12:00:263
739499118,2cyclictest7242-21grep12:25:243
739499118,2cyclictest4051-21cut11:45:183
739499118,2cyclictest3556-21grep08:05:253
739499118,2cyclictest3556-21grep08:05:243
739499118,2cyclictest32401-21ssh10:06:223
739499118,2cyclictest29748-21tr07:55:143
739499118,2cyclictest29070-21expr09:00:143
739499118,2cyclictest2835-21mailstats09:10:223
739499118,2cyclictest27550-21ssh10:23:083
739499118,2cyclictest27460-21ssh11:01:023
739499118,2cyclictest24565-21cat07:45:003
739499118,2cyclictest18168-21ssh09:20:093
739499118,2cyclictest11647-21grep09:15:243
739499117,3cyclictest19044-21sed07:30:253
739499117,2cyclictest20525-21head07:35:163
739499110,11cyclictest0-21swapper/310:56:183
739499110,10cyclictest21196-21chrt12:15:153
739499110,10cyclictest0-21swapper/307:50:123
739199118,2cyclictest31312-21grep12:20:262
739199118,2cyclictest30816-21ssh09:46:072
739199118,2cyclictest27124-21cut08:55:152
739199118,2cyclictest25243-21wc08:50:222
739199118,2cyclictest20786-21grep11:55:162
739199118,2cyclictest19624-21ls08:40:152
739199118,2cyclictest1678-21ssh11:25:032
739199118,2cyclictest14433-21tr08:30:142
739199118,2cyclictest14423-21taskset07:24:102
739199118,2cyclictest1375-21sed08:00:262
739199118,2cyclictest12453-21cut09:55:132
739199118,2cyclictest11537-21cat08:24:552
739199117,3cyclictest23891-21grep10:40:142
739199117,2cyclictest24809-21cut07:45:142
739199110,0cyclictest0-21swapper/209:15:412
738899118,2cyclictest26730-21expr08:55:131
738899117,3cyclictest10916-21/usr/sbin/munin07:15:171
738899117,3cyclictest0-21swapper/108:20:221
738899114,4cyclictest0-21swapper/108:40:231
738599118,2cyclictest9008-21ssh11:48:240
738599118,2cyclictest8775-21ssh09:14:360
738599118,2cyclictest8541-21ssh10:30:280
738599118,2cyclictest4404-21wc08:10:010
738599118,2cyclictest4404-21wc08:10:000
738599118,2cyclictest3614-21ssh10:47:030
738599118,2cyclictest32738-21ssh09:28:350
738599118,2cyclictest30769-21ssh10:44:290
738599118,2cyclictest26329-21rm11:00:240
738599118,2cyclictest2511-21ssh09:30:060
738599118,2cyclictest24998-21ssh12:16:570
738599118,2cyclictest21657-21taskset07:37:030
738599118,2cyclictest18547-21ssh09:39:400
738599118,2cyclictest18450-21ssh10:55:340
738599118,2cyclictest17695-21ssh12:31:520
738599118,2cyclictest17591-21ssh09:57:550
738599118,2cyclictest14657-21ssh10:35:010
738599118,2cyclictest14331-21cut08:30:130
738599118,2cyclictest13968-21wc07:20:250
738599118,2cyclictest13352-21ssh11:12:130
738599118,2cyclictest11550-21sh08:25:010
738599117,3cyclictest9995-21tr09:15:130
738599116,3cyclictest17553-21grep07:30:120
738599111,7cyclictest0-21swapper/007:50:120
738599110,3cyclictest0-21swapper/012:35:250
738599110,1cyclictest0-21swapper/007:13:350
739499108,1cyclictest9523-21awk08:20:143
739499108,1cyclictest65892sleep308:12:443
739499108,1cyclictest31503-21cat09:05:143
739499108,1cyclictest259582sleep308:53:263
739499108,1cyclictest16837-21apache_accesses08:35:113
739499108,1cyclictest11959-21taskset07:18:153
739499107,2cyclictest11817-21apache_accesses08:25:133
739199108,1cyclictest289922sleep207:51:462
739199108,1cyclictest22426-21basename08:45:172
739199108,1cyclictest22233-21basename07:40:152
739199108,1cyclictest18718-21taskset08:37:092
739199107,2cyclictest7629-21chrt08:15:152
739199107,2cyclictest28850-21grep09:00:142
739199106,3cyclictest17905-21df07:30:142
739199104,4cyclictest19791-21/usr/sbin/munin07:35:122
739199100,9cyclictest4511-21cat08:10:002
738899107,2cyclictest29706-21grep07:55:141
738899107,2cyclictest27930-21head07:50:161
738899107,2cyclictest12819-21cat07:20:151
738899104,3cyclictest0-21swapper/107:30:191
738899104,2cyclictest452-21systemd-logind09:01:561
738899100,9cyclictest20223-21expr07:35:151
738599107,2cyclictest32752-21cut08:00:170
738599107,2cyclictest28975-21grep09:00:140
738599107,2cyclictest15074-21cut07:25:140
738599101,5cyclictest225-21systemd-journal08:55:000
73949996,2cyclictest22641-21ls08:45:203
73949993,2cyclictest225-21systemd-journal07:25:003
73919991,2cyclictest225-21systemd-journal07:17:352
73859996,2cyclictest17033-21ls08:35:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional