You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 13:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 10, 2025 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16806210981,9sleep30-21swapper/319:07:263
1674729985,9sleep10-21swapper/119:06:391
1668328674,8sleep00-21swapper/019:05:490
1690728567,14sleep20-21swapper/219:08:452
65872710,2sleep230043-21kworker/2:221:53:522
1708999260,2cyclictest0-21swapper/123:50:121
17093992522,2cyclictest32-21ksoftirqd/223:59:592
1709899240,23cyclictest0-21swapper/300:05:143
1709399233,19cyclictest18952-21munin-run21:40:002
197252220,0sleep00-21swapper/023:59:290
1709399220,22cyclictest0-21swapper/223:38:082
17089992221,1cyclictest25-21ksoftirqd/100:28:261
30722210,1sleep10-21swapper/100:35:161
17098992120,1cyclictest0-21swapper/320:20:163
1709899210,5cyclictest141rcu_preempt23:10:143
17093992118,2cyclictest0-21swapper/222:10:202
1708199202,17cyclictest0-21swapper/020:40:130
1709399192,2cyclictest141rcu_preempt22:22:452
1709399190,19cyclictest0-21swapper/222:19:332
1708999190,18cyclictest0-21swapper/123:20:121
1708999190,18cyclictest0-21swapper/123:20:111
1708199196,6cyclictest3959-21munin-run19:50:000
17093991817,1cyclictest0-21swapper/220:25:212
1708999180,0cyclictest0-21swapper/122:45:321
1709899172,1cyclictest381rcuc/322:47:213
1709899171,2cyclictest39-21ksoftirqd/322:50:273
1709899171,2cyclictest0-21swapper/323:30:533
1709899170,2cyclictest0-21swapper/322:08:253
17089991716,1cyclictest0-21swapper/100:05:141
1708999170,2cyclictest0-21swapper/122:39:301
17081991714,2cyclictest1962-21sh23:23:220
17081991714,2cyclictest1962-21sh23:23:210
87722160,0chrt0-21swapper/123:28:081
1709899169,4cyclictest0-21swapper/300:10:173
1709899166,2cyclictest39-21ksoftirqd/321:57:143
1709899164,9cyclictest39-21ksoftirqd/323:08:063
1709899162,12cyclictest39-21ksoftirqd/321:16:343
17098991614,1cyclictest0-21swapper/321:00:163
17098991610,5cyclictest5735-21cut21:30:183
17098991610,5cyclictest5735-21cut21:30:173
1709399161,3cyclictest7744-21snmpd21:46:272
17093991612,3cyclictest31358-21ssh00:07:052
1708999160,0cyclictest0-21swapper/121:44:311
17081991613,2cyclictest13-21ksoftirqd/021:45:160
17098991513,2cyclictest39-21ksoftirqd/300:18:433
17098991513,1cyclictest381rcuc/300:01:153
1709899150,2cyclictest0-21swapper/321:12:223
17093991512,2cyclictest0-21swapper/200:25:252
1709399151,1cyclictest141rcu_preempt23:25:022
17093991511,3cyclictest0-21swapper/221:23:372
1709399150,1cyclictest0-21swapper/223:30:432
1708999150,2cyclictest0-21swapper/122:08:121
1708199158,4cyclictest9885-21diskmemload22:08:540
1708199156,8cyclictest21217-21munin-run22:05:010
1708199150,0cyclictest0-21swapper/023:54:160
1709899148,5cyclictest0-21swapper/322:13:123
1709899147,6cyclictest318-21unixbench_multi21:25:263
1709899142,10cyclictest0-21swapper/300:27:253
1709899141,3cyclictest0-21swapper/319:40:103
17098991410,3cyclictest13853-21tr21:35:233
1709899140,3cyclictest0-21swapper/300:31:443
1709899140,2cyclictest0-21swapper/323:53:053
1709399147,6cyclictest22663-21sh22:28:392
1709399142,11cyclictest11083-21cron20:05:012
17093991413,1cyclictest29112-21grep22:55:302
17093991411,2cyclictest23542-21grep22:05:272
17093991410,3cyclictest29259-21grep19:35:002
1709399140,1cyclictest0-21swapper/200:23:352
1709399140,14cyclictest0-21swapper/221:55:162
1709399140,14cyclictest0-21swapper/200:16:582
17089991413,1cyclictest0-21swapper/122:29:071
17089991410,3cyclictest14489-21cut20:10:161
17089991410,2cyclictest14421-21grep21:35:281
17081991410,3cyclictest22023-21cron19:20:000
17081991410,3cyclictest16777-21cron22:24:590
1709899139,3cyclictest26076-21sh21:45:003
1709899139,2cyclictest0-21swapper/323:00:133
1709899137,5cyclictest39-21ksoftirqd/323:29:593
1709899137,4cyclictest381rcuc/322:37:333
1709899136,3cyclictest141rcu_preempt22:41:043
1709899135,8cyclictest0-21swapper/300:35:143
1709899135,6cyclictest9896-21munin-run21:10:003
1709899134,3cyclictest39-21ksoftirqd/323:56:453
1709899131,2cyclictest141rcu_preempt22:04:453
17098991311,2cyclictest17079-21cyclictest22:20:183
17098991310,2cyclictest7547-21cat19:55:203
17098991310,2cyclictest703-21sh21:50:013
17098991310,2cyclictest3351-21cat19:45:283
17098991310,2cyclictest26209-21head23:40:183
17098991310,2cyclictest25386-21grep22:30:213
17098991310,2cyclictest25035-21grep19:25:163
1709899130,12cyclictest16211-21grep20:15:163
1709399139,3cyclictest9192-21wc21:05:262
1709399139,3cyclictest32122-21wc20:45:252
1709399138,3cyclictest37850irq/131-snd_hda22:35:542
1709399135,4cyclictest27265-21kworker/u8:223:10:522
17093991313,0cyclictest0-21swapper/221:14:182
17093991310,2cyclictest8697-21apt20:00:002
17093991310,2cyclictest6475-21grep21:30:222
17093991310,2cyclictest6475-21grep21:30:212
17093991310,2cyclictest28006-21cron20:39:592
17093991310,2cyclictest20368-21grep20:20:292
17093991310,2cyclictest15125-21basename22:00:192
17093991310,2cyclictest13154-21cut22:45:142
17093991310,2cyclictest1233-21cut20:50:162
17093991310,2cyclictest0-21swapper/200:05:002
1709399130,13cyclictest0-21swapper/222:30:062
1709399130,13cyclictest0-21swapper/200:36:332
1709399130,12cyclictest0-21swapper/221:42:162
1708999132,2cyclictest224-21systemd-journal23:41:501
17089991310,2cyclictest30406-21sh20:44:591
17089991310,2cyclictest2458-21head23:00:171
17089991310,2cyclictest24560-21wc20:30:201
17089991310,2cyclictest15820-21tr23:10:141
17089991310,2cyclictest11432-21cat22:20:211
1708999130,13cyclictest0-21swapper/121:26:261
1708999130,12cyclictest751-21cut23:45:211
1708999130,12cyclictest4259-21wc20:55:231
1708199139,3cyclictest29616-21cut00:30:190
1708199139,3cyclictest28494-21grep00:05:180
1708199139,3cyclictest18543-21ssh21:15:220
1708199139,3cyclictest16025-21ssh00:20:150
1708199131,6cyclictest5125-21munin-run21:00:000
17081991312,1cyclictest0-21swapper/021:20:180
17081991310,2cyclictest4609-21grep23:25:140
17081991310,2cyclictest24155-21grep20:30:170
17081991310,2cyclictest22013-21munin-run23:15:010
17081991310,2cyclictest17819-21grep19:10:150
17081991310,2cyclictest12166-21cat21:35:000
17081991310,2cyclictest12166-21cat21:35:000
1709899129,2cyclictest39-21ksoftirqd/322:15:223
1709899129,2cyclictest32457-21grep23:45:183
1709899129,2cyclictest31451-21sed20:45:173
1709899129,2cyclictest23352-21grep23:15:213
1709899128,3cyclictest3694-21grep20:55:183
1709899128,2cyclictest381rcuc/323:35:013
1709899127,2cyclictest224-21systemd-journal19:25:013
1709399129,2cyclictest9353-21grep23:05:182
1709399129,2cyclictest7805-21wc23:50:172
1709399129,2cyclictest5152-21sleep21:30:002
1709399129,2cyclictest3514-21grep19:45:282
1709399129,2cyclictest30833-21chrt23:43:562
1709399129,2cyclictest30423-21perf20:45:002
1709399129,2cyclictest23726-21tr20:30:142
1709399129,2cyclictest17819-21grep20:15:292
1709399129,2cyclictest14488-21sed20:10:162
1709399128,3cyclictest12615-21sleep22:45:012
1709399120,11cyclictest8873-21wc00:15:002
1708999129,2cyclictest7553-21grep21:30:301
1708999129,2cyclictest7553-21grep21:30:301
1708999129,2cyclictest6866-21grep22:40:191
1708999129,2cyclictest31878-21cut20:45:221
1708999129,2cyclictest27306-21cut22:55:181
1708999129,2cyclictest20340-21tr22:50:151
1708999129,2cyclictest20214-21sh23:59:591
1708999128,3cyclictest7499-21cron21:05:001
1708999124,4cyclictest224-21systemd-journal00:00:011
1708999122,6cyclictest2199-21ssh21:50:191
1708999122,4cyclictest0-21swapper/119:45:001
1708999120,0cyclictest0-21swapper/120:00:151
1708199129,2cyclictest2682-21grep00:10:160
1708199129,2cyclictest24977-21sleep23:40:010
1708199129,2cyclictest20388-21tr22:50:150
1708199129,2cyclictest1916-21sh21:26:520
1708199129,2cyclictest14833-21sort22:45:270
1708199129,2cyclictest14654-21cat22:00:000
1708199129,2cyclictest12597-21apt22:45:000
1708199129,2cyclictest10454-21tr00:15:220
1708199128,2cyclictest0-21swapper/023:16:170
1708199126,5cyclictest0-21swapper/019:20:150
1708199121,11cyclictest0-21swapper/022:59:510
1708199120,1cyclictest0-21swapper/023:48:390
1709899118,2cyclictest5408-21ls19:50:213
1709899118,2cyclictest5014-21sh21:52:183
1709899118,2cyclictest30201-21ssh23:20:173
1709899118,2cyclictest30201-21ssh23:20:163
1709899118,2cyclictest29275-21ssh22:55:353
1709899118,2cyclictest2797-21date20:55:003
1709899118,2cyclictest26893-21cat20:35:193
1709899118,2cyclictest20056-21tr19:15:153
1709899118,2cyclictest17381-21cat19:10:003
1709899118,2cyclictest17288-21rm22:25:043
1709899118,2cyclictest15864-21cat00:20:153
17098991110,1cyclictest39-21ksoftirqd/319:30:193
1709399119,1cyclictest32675-21tr23:45:192
1709399118,2cyclictest32710-21ssh23:21:422
1709399118,2cyclictest32710-21ssh23:21:422
1709399118,2cyclictest30470-21cat19:35:232
1709399118,2cyclictest29256-21grep00:30:162
1709399118,2cyclictest23370-21head23:15:212
1709399118,2cyclictest22322-21chrt19:20:152
1709399117,1cyclictest224-21systemd-journal19:15:002
1708999118,2cyclictest9973-21cut00:15:191
1708999118,2cyclictest6859-21grep19:55:151
1708999118,2cyclictest6677-21ssh22:16:551
1708999118,2cyclictest4508-21cstates19:50:151
1708999118,2cyclictest32298-21ssh00:31:591
1708999118,2cyclictest30790-21ssh22:10:291
1708999118,2cyclictest28226-21grep19:30:221
1708999118,2cyclictest28080-21tr21:45:251
1708999118,2cyclictest27767-21sh22:31:421
1708999118,2cyclictest27407-21sed20:35:261
1708999118,2cyclictest26414-21ssh21:20:561
1708999118,2cyclictest23990-21grep19:20:271
1708999118,2cyclictest2295-21ssh00:10:081
1708999118,2cyclictest21816-21tail20:25:181
1708999118,2cyclictest20810-21ssh23:36:031
1708999118,2cyclictest17664-21ssh22:01:151
1708999118,2cyclictest16309-21tail00:20:171
1708999118,2cyclictest11221-21ssh23:30:141
1708999118,2cyclictest10731-21ssh21:10:191
1708999117,3cyclictest9615-21wc23:05:201
1708199119,1cyclictest18941-21sh21:40:010
1708199118,2cyclictest9669-21seq21:07:260
1708199118,2cyclictest8705-21basename20:00:000
1708199118,2cyclictest3284-21rm22:15:150
1708199118,2cyclictest32599-21head19:40:130
1708199118,2cyclictest31872-21ssh22:11:330
1708199118,2cyclictest31672-21cat20:45:200
1708199118,2cyclictest27922-21chrt20:38:570
1708199118,2cyclictest27267-21sed23:40:270
1708199118,2cyclictest2722-21sh00:35:030
1708199118,2cyclictest25019-21tr19:25:160
1708199118,2cyclictest2464-21head23:00:170
1708199118,2cyclictest22714-21sh21:41:460
1708199118,2cyclictest22386-21tr00:25:160
1708199118,2cyclictest21587-21cat00:00:190
1708199118,2cyclictest21240-21wget20:25:150
1708199118,2cyclictest19648-21ssh22:25:380
1708199118,2cyclictest13421-21chrt20:09:330
1708199118,2cyclictest12755-21ssh23:07:400
1708199118,2cyclictest1136-21acpi21:50:120
1708199118,2cyclictest11180-21grep23:30:130
1708199112,6cyclictest19443-21gltestperf20:20:190
1708199110,9cyclictest0-21swapper/019:50:140
1708199110,10cyclictest5701-21cat22:40:000
1709899108,1cyclictest21645-21df_abs20:25:173
1709899108,1cyclictest15679-21seq20:13:053
1709899108,1cyclictest11670-21wc20:05:143
1709899107,2cyclictest29560-21cut19:35:163
1709899107,2cyclictest23924-21expr20:30:143
1709899107,2cyclictest10843-21systemd-cgroups20:01:573
1709899100,9cyclictest25173-21cat21:20:203
1709399108,1cyclictest73932sleep221:03:432
1709399108,1cyclictest24813-21cut19:25:152
1709399108,1cyclictest20492-21df_inode19:15:162
1709399108,1cyclictest19998-21ssh22:50:102
1709399107,2cyclictest5309-21cat19:50:192
1709399107,2cyclictest32072-21egrep19:40:112
1709399107,2cyclictest2253-21sed23:00:152
1709399107,2cyclictest17463-21tr21:15:152
1709399101,8cyclictest5149-21perf21:00:012
1708999108,1cyclictest313182sleep119:36:291
1708999108,1cyclictest21432-21sort19:15:261
1708999108,1cyclictest18839-21sh21:15:251
1708999108,1cyclictest17787-21expr19:10:141
1708999107,2cyclictest26800-21chrt19:29:261
1708999107,2cyclictest23408-21ssh23:15:211
1708999107,2cyclictest20358-21sed20:20:291
1708999107,2cyclictest2030-21grep19:45:151
1708999107,2cyclictest16063-21chrt20:15:031
1708999107,2cyclictest10848-21ssh21:56:141
1708999104,4cyclictest1104-21sed20:50:151
1708999100,9cyclictest13195-21taskset20:06:371
1708199108,1cyclictest27147-21cut19:30:160
1708199108,1cyclictest1531-21cat20:50:180
1708199108,1cyclictest11267-21hwlatdetect21:10:220
1708199108,1cyclictest10801-21chrt20:01:230
1708199107,2cyclictest5383-21bc21:00:140
1708199107,2cyclictest14449-21expr20:10:160
1708199103,6cyclictest24568-21/usr/sbin/munin22:30:210
1708199100,9cyclictest29463-21taskset19:35:120
1708199100,9cyclictest16981-21if_enp2s020:15:210
170989991,5cyclictest28334-21grep20:40:143
170939996,2cyclictest12236-21if_err_enp0s31f20:05:182
170899997,1cyclictest7786-21cut21:05:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional