You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 03:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Jan 05, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1487228875,9sleep20-21swapper/219:09:072
1484528875,9sleep00-21swapper/019:08:450
1476528859,9sleep30-21swapper/319:07:433
1362728063,13sleep10-21swapper/119:05:211
1503299291,21cyclictest225-21systemd-journal22:26:240
1504199250,10cyclictest0-21swapper/320:45:123
1503299231,21cyclictest20273-21tr21:35:150
1503299230,22cyclictest0-21swapper/023:05:150
1503299230,22cyclictest0-21swapper/000:30:160
1503299230,22cyclictest0-21swapper/000:30:150
1503299230,21cyclictest0-21swapper/023:30:150
15038992221,0cyclictest0-21swapper/221:05:182
1503599220,21cyclictest0-21swapper/120:50:141
1503299220,21cyclictest21843-21fschecks_count20:30:170
1503899210,1cyclictest27719-21users20:40:282
1503299204,8cyclictest7736-21idleruntime-cro00:30:000
1503299201,16cyclictest29409-21apt-config21:55:000
1503299200,14cyclictest0-21swapper/000:00:160
1504199190,14cyclictest0-21swapper/322:40:163
1503299196,7cyclictest9008-21awk20:05:000
1503299191,17cyclictest19969-21sh19:20:000
1503299191,17cyclictest0-21swapper/022:35:140
1503299191,17cyclictest0-21swapper/022:35:130
1503299191,17cyclictest0-21swapper/022:20:130
1503299190,16cyclictest0-21swapper/021:25:130
1503899183,12cyclictest477-21rs:main2
15035991818,0cyclictest0-21swapper/123:45:131
1503299181,12cyclictest225-21systemd-journal20:41:320
1503299180,17cyclictest0-21swapper/022:05:000
1503899173,3cyclictest15158-21latency_hist21:24:592
1503299179,7cyclictest7851-21chrt21:10:120
1503299170,16cyclictest0-21swapper/021:05:180
1503299170,15cyclictest0-21swapper/021:20:140
1503299170,12cyclictest0-21swapper/020:05:140
1504199165,10cyclictest13895-21apache223:30:213
1503299169,6cyclictest16379-21sed20:20:000
1503299168,7cyclictest22971-21cron23:55:010
1503299168,6cyclictest1629-21tail23:10:230
1503299161,13cyclictest13497-21taskset20:12:040
1503299160,14cyclictest0-21swapper/019:10:160
1503299160,13cyclictest0-21swapper/021:15:140
15041991512,2cyclictest39-21ksoftirqd/320:42:343
1504199150,0cyclictest0-21swapper/300:35:163
1503599156,5cyclictest225-21systemd-journal21:00:001
1503299158,6cyclictest17158-21cut20:20:190
1503299154,8cyclictest13951-21apache219:20:130
1503299151,13cyclictest3046-21kernelversion19:50:240
1503299151,11cyclictest30471-21gltestperf21:55:180
1503299151,11cyclictest0-21swapper/021:40:120
1503299150,15cyclictest0-21swapper/020:55:180
1503299150,14cyclictest0-21swapper/022:30:220
1503299150,12cyclictest0-21swapper/023:35:130
15041991411,2cyclictest18386-21grep23:45:143
15041991411,2cyclictest16957-21sed19:10:263
15041991411,2cyclictest13764-21awk22:30:003
15038991411,2cyclictest22176-21sendmail-msp21:39:592
15035991410,3cyclictest29580-21wc19:39:591
1503299149,3cyclictest25553-21cut21:45:170
1503299149,3cyclictest25441-21chrt19:30:160
1503299148,5cyclictest888-21cut00:15:150
1503299148,5cyclictest8707-21taskset22:16:430
1503299148,5cyclictest5933-21cut23:20:160
1503299148,5cyclictest325-21cut19:45:150
1503299148,5cyclictest24205-21ls19:25:300
1503299148,5cyclictest23246-21taskset23:55:120
1503299148,5cyclictest21634-21sort22:45:160
1503299148,5cyclictest18075-21cat23:45:000
1503299147,6cyclictest4188-21grep00:20:200
1503299147,6cyclictest28978-21expr23:00:150
1503299147,6cyclictest27641-21cat22:55:240
1503299147,6cyclictest24032-21tr22:50:130
1503299143,5cyclictest30496-21perf20:50:000
1503299142,11cyclictest30919-21awk19:40:220
1503299141,11cyclictest19989-21awk23:45:270
1503299140,13cyclictest18052-21cat21:30:160
1503299140,13cyclictest0-21swapper/021:00:190
1503299140,12cyclictest0-21swapper/020:50:130
1503299140,11cyclictest225-21systemd-journal19:36:220
99712130,1chrt9969-21grep21:15:013
1504199139,3cyclictest4326-21wc19:55:003
1504199139,3cyclictest3818-21sed22:05:283
1504199139,3cyclictest14306-21tr22:30:143
1504199139,2cyclictest570-21cron00:15:003
1504199131,7cyclictest15201-21cron23:40:003
15041991310,2cyclictest9980-21head20:05:183
15041991310,2cyclictest6396-21grep00:25:173
15041991310,2cyclictest28907-21sort00:05:173
15041991310,2cyclictest24815-21cat23:55:273
15041991310,2cyclictest23085-21grep22:45:273
15041991310,2cyclictest21218-21sort19:20:213
15041991310,2cyclictest17244-21grep20:20:203
15041991310,2cyclictest1350-21grep22:00:273
15041991310,2cyclictest12953-21fgrep20:10:253
1504199130,12cyclictest18347-21grep19:15:183
1503899139,3cyclictest18419-21grep21:30:172
15038991310,2cyclictest9860-21grep22:20:172
15038991310,2cyclictest5392-21cat00:25:002
15038991310,2cyclictest5159-21grep19:55:192
15038991310,2cyclictest2798-21cron21:00:002
15038991310,2cyclictest26124-21tr00:00:182
15038991310,2cyclictest18503-21grep19:15:202
15038991310,2cyclictest17399-21munin-run21:30:002
15038991310,2cyclictest11362-21apt20:10:012
1503899130,12cyclictest8788-21grep00:30:202
1503899130,12cyclictest8788-21grep00:30:192
1503899130,12cyclictest12097-21grep00:35:292
1503599139,3cyclictest25481-21grep22:50:251
1503599139,2cyclictest9289-21grep20:05:141
1503599139,2cyclictest21029-21grep19:20:191
1503599139,2cyclictest18592-21grep19:15:201
1503599139,2cyclictest17454-21sort19:15:001
1503599138,4cyclictest13027-21taskset22:25:251
1503599138,3cyclictest23410-21head19:25:211
15035991310,2cyclictest7649-21cut20:00:181
15035991310,2cyclictest3208-21cat23:15:001
15035991310,2cyclictest32029-21tr00:10:201
15035991310,2cyclictest27103-21cron19:35:001
15035991310,2cyclictest15952-21tr23:40:161
15035991310,2cyclictest13096-21tr21:20:151
1503599130,12cyclictest24688-21grep20:35:211
1503299136,6cyclictest5223-21tail19:55:190
1503299131,11cyclictest0-21swapper/022:05:120
1503299131,10cyclictest10319-21latency_hist23:30:010
1503299130,12cyclictest20864-21taskset22:41:140
1503299130,12cyclictest0-21swapper/020:25:210
1503299130,12cyclictest0-21swapper/000:10:260
1503299130,10cyclictest4461-21/usr/sbin/munin22:10:290
1504199129,2cyclictest820-21grep19:45:183
1504199129,2cyclictest8108-21grep20:00:233
1504199129,2cyclictest5085-21sort19:55:193
1504199129,2cyclictest32077-21wc22:00:003
1504199129,2cyclictest28869-21grep23:00:153
1504199129,2cyclictest28293-21sh23:00:013
1504199129,2cyclictest28280-21cut21:50:213
1504199129,2cyclictest24048-21grep21:40:283
1504199129,2cyclictest21164-21tr21:35:223
1504199129,2cyclictest19551-21head20:25:173
1504199129,2cyclictest13785-21cut21:20:203
1504199129,2cyclictest11102-21tune2fs21:15:173
1504199125,4cyclictest27385-21apt-get19:35:143
1503899129,2cyclictest9329-21cat21:10:262
1503899129,2cyclictest2996-21readlink00:20:012
1503899129,2cyclictest27928-21sed21:50:182
1503899129,2cyclictest25655-21sort19:30:172
1503899129,2cyclictest24586-21sed22:50:162
1503899129,2cyclictest22514-21cut21:40:152
1503899129,2cyclictest19628-21head20:25:172
1503899129,2cyclictest15681-21sed20:15:302
1503899129,2cyclictest11852-21cat21:15:272
1503899126,2cyclictest225-21systemd-journal22:15:002
1503899124,2cyclictest225-21systemd-journal23:40:002
1503899123,1cyclictest225-21systemd-journal00:07:342
1503899120,11cyclictest15979-21head19:10:162
1503599129,2cyclictest8612-21grep23:25:171
1503599129,2cyclictest6585-21cron20:00:001
1503599129,2cyclictest4721-21grep21:00:301
1503599129,2cyclictest32226-21cut22:00:141
1503599129,2cyclictest31919-21wc23:05:211
1503599129,2cyclictest29989-21sed20:45:251
1503599129,2cyclictest29797-21tr21:55:151
1503599129,2cyclictest26859-21cut22:55:161
1503599129,2cyclictest22195-21wc23:50:241
1503599129,2cyclictest21732-21wc22:45:161
1503599129,2cyclictest21531-21wc20:30:161
1503599129,2cyclictest15725-21tail22:30:271
1503599129,2cyclictest14303-21tr23:35:231
1503599128,3cyclictest32389-21tr19:45:121
1503599127,3cyclictest0-21swapper/121:40:141
1503599120,12cyclictest0-21swapper/121:25:391
1503599120,11cyclictest18585-21grep20:25:011
1503299128,3cyclictest51702sleep023:17:100
1503299126,5cyclictest29651-21sort00:05:270
1503299126,5cyclictest24592-21sed20:35:200
1503299125,6cyclictest11434-21wc00:35:220
1504199118,2cyclictest8771-21head00:30:193
1504199118,2cyclictest8771-21head00:30:193
1504199118,2cyclictest3762-21cut21:00:193
1504199118,2cyclictest31223-21grep23:05:163
1504199118,2cyclictest30225-21cat19:40:183
1504199118,2cyclictest2910-21taskset00:19:263
1504199118,2cyclictest26105-21tr00:00:183
1504199118,2cyclictest25920-21cut20:40:003
1504199118,2cyclictest25817-21wc19:30:183
1504199118,2cyclictest25018-21tr21:45:143
1504199118,2cyclictest22628-21apache_accesses19:25:163
1504199118,2cyclictest21694-21expr20:30:163
1504199118,2cyclictest18697-21taskset22:39:103
1504199118,2cyclictest18697-21taskset22:39:103
1504199118,2cyclictest1646-21grep23:10:223
1504199118,2cyclictest10164-21cat22:20:193
1504199110,10cyclictest20690-21dpkg23:50:003
1504199110,10cyclictest2022-21sendmail_mailst20:55:233
1503899119,1cyclictest3806-21grep19:50:322
1503899119,1cyclictest29309-21head23:00:172
1503899118,2cyclictest8003-21awk20:00:212
1503899118,2cyclictest6749-21ls23:20:222
1503899118,2cyclictest471-21head19:45:152
1503899118,2cyclictest3200-21tr21:00:162
1503899118,2cyclictest31137-21expr20:50:152
1503899118,2cyclictest30454-21cut19:40:192
1503899118,2cyclictest28908-21df20:45:142
1503899118,2cyclictest26406-21tr22:55:132
1503899118,2cyclictest25698-21cron20:40:012
1503899118,2cyclictest24233-21cat23:55:192
1503899118,2cyclictest2367-21wc23:10:322
1503899118,2cyclictest22921-21tr19:25:182
1503899118,2cyclictest15795-21grep22:30:272
1503899118,2cyclictest11989-21expr22:25:152
1503899111,1cyclictest225-21systemd-journal00:10:032
1503899110,10cyclictest28207-21cat19:35:182
1503899110,10cyclictest15511-21apt-key23:40:002
1503599118,2cyclictest9533-21grep22:20:151
1503599118,2cyclictest7487-21cut22:15:161
1503599118,2cyclictest7385-21grep23:20:281
1503599118,2cyclictest6099-21sort21:05:211
1503599118,2cyclictest5153-21taskset00:22:291
1503599118,2cyclictest4738-21grep22:10:171
1503599118,2cyclictest3327-21grep22:05:221
1503599118,2cyclictest31563-21taskset19:41:001
1503599118,2cyclictest26578-21grep00:00:201
1503599118,2cyclictest2422-21chrt19:50:201
1503599118,2cyclictest23493-21tr23:55:151
1503599118,2cyclictest1713-21cat00:15:201
1503599118,2cyclictest16926-21tr22:35:161
1503599118,2cyclictest16926-21tr22:35:151
1503599118,2cyclictest16006-21taskset20:17:531
1503599118,2cyclictest10626-21grep00:35:161
1503599118,2cyclictest10391-21taskset23:30:001
1503599117,3cyclictest22139-21sh21:39:591
1504199108,1cyclictest98302sleep323:25:463
1504199108,1cyclictest5934-21expr23:20:173
1504199108,1cyclictest4680-21egrep22:10:163
1504199108,1cyclictest325572sleep320:51:243
1504199108,1cyclictest15974-21taskset20:17:293
1504199107,2cyclictest8887-21taskset22:19:003
1504199107,2cyclictest5702-21tr21:05:193
1504199107,2cyclictest5158-21chrt23:17:013
1504199107,2cyclictest3683-21expr00:20:163
1504199107,2cyclictest18004-21cat21:30:153
1504199107,2cyclictest15575-21cat21:25:163
1504199100,9cyclictest15528-21mktemp23:40:003
1503899108,2cyclictest8156-21basename23:25:162
1503899108,1cyclictest7258-21expr22:15:142
1503899108,1cyclictest3975-21chrt22:06:572
1503899108,1cyclictest30595-21basename21:55:192
1503899108,1cyclictest21383-21ntpq19:20:232
1503899108,1cyclictest20833-21chrt23:50:062
1503899107,2cyclictest31711-21expr23:05:192
1503899107,2cyclictest19120-21aten2_r7power_e22:40:172
1503899107,2cyclictest16853-21cpu22:35:162
1503899107,2cyclictest16853-21cpu22:35:152
1503899107,2cyclictest12296-21seq23:31:562
1503899106,3cyclictest13927-21apache223:45:192
1503899102,4cyclictest225-21systemd-journal22:05:002
1503599108,1cyclictest98402sleep100:31:411
1503599108,1cyclictest98402sleep100:31:401
1503599108,1cyclictest9291-21wc21:10:261
1503599108,1cyclictest7508-21taskset00:27:141
1503599108,1cyclictest29513-21kernelversion23:00:191
1503599108,1cyclictest290432sleep121:52:041
1503599108,1cyclictest27862-21taskset20:42:071
1503599108,1cyclictest19090-21cut22:40:161
1503599108,1cyclictest13399-21taskset20:10:491
1503599107,2cyclictest29486-21wc00:05:221
1503599107,2cyclictest25422-21sort21:45:161
1503599107,1cyclictest10698-21cut21:15:151
1503599105,2cyclictest225-21systemd-journal23:17:341
1503599102,4cyclictest225-21systemd-journal20:29:551
1503599101,3cyclictest225-21systemd-journal21:31:411
150419996,2cyclictest25521-21sed22:50:253
150389998,1cyclictest16844-21expr20:20:182
150389997,1cyclictest5095-21taskset23:16:172
150389997,1cyclictest24877-21basename21:45:142
150389997,1cyclictest21745-21cat22:45:152
150389997,1cyclictest11694-21hostname20:10:162
150389996,2cyclictest21959-21expr20:30:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional