You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-18 - 16:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Fri Apr 18, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
788699250,24cyclictest27588-21latency_hist07:50:002
788299250,24cyclictest13032-21sendmail07:20:001
787999240,23cyclictest2971-21idleruntime-cro11:50:000
788999230,22cyclictest5554-21cron10:35:013
7886992321,1cyclictest7142-21cron08:15:002
788699230,22cyclictest771-21munin-run10:25:012
788699230,22cyclictest0-21swapper/208:03:282
7882992321,1cyclictest1093-21apt-get11:48:251
787999230,22cyclictest22695-21gzip07:39:590
787999230,22cyclictest19134-21awk12:29:590
788999220,21cyclictest6202-21idleruntime-cro10:40:003
788999220,21cyclictest4129-21apt-get09:18:253
788999220,21cyclictest30584-21awk11:40:003
788999220,21cyclictest0-21swapper/308:28:263
788999220,1cyclictest32418-21awk08:00:013
7886992222,0cyclictest0-21swapper/211:03:202
7886992220,1cyclictest9587-21apt08:20:002
7886992220,1cyclictest8638-21munin-run10:45:002
7886992220,1cyclictest19712-21perf11:10:012
788699220,22cyclictest0-21swapper/210:03:212
788699220,21cyclictest0-21swapper/209:48:282
788699220,1cyclictest8411-21cron12:05:002
788699220,1cyclictest28847-21grep09:00:012
788699220,1cyclictest11166-21apt-get07:18:282
7882992221,0cyclictest0-21swapper/107:38:281
7882992220,1cyclictest612-21gdbus11:40:001
7882992220,1cyclictest10842-21munin-run12:10:011
7882992220,1cyclictest0-21swapper/111:44:591
7882992220,1cyclictest0-21swapper/111:00:021
7882992220,1cyclictest0-21swapper/108:50:011
7882992220,1cyclictest0-21swapper/107:55:001
7882992220,1cyclictest0-21swapper/107:55:001
788299220,21cyclictest28255-21apt-get10:13:281
788299220,1cyclictest22290-21awk09:54:591
788299220,1cyclictest22094-21perf11:15:011
788299220,1cyclictest0-21swapper/108:23:301
7879992222,0cyclictest0-21swapper/008:18:260
7879992220,1cyclictest6024-21basename12:00:010
7879992220,1cyclictest20294-21awk07:35:000
787999220,21cyclictest598-21rs:main0
787999220,21cyclictest28801-21cron09:00:010
787999220,21cyclictest25670-21apt-get11:28:250
787999220,21cyclictest20225-21apt-get11:13:280
787999220,21cyclictest1285-21apt-get10:28:260
787999220,21cyclictest11166-21apt-get07:18:280
787999220,21cyclictest0-21swapper/009:05:000
7889992120,0cyclictest19766-21apt-get08:43:283
7889992119,1cyclictest19237-21apt08:40:003
788999210,21cyclictest0-21swapper/309:43:223
788999210,20cyclictest0-21swapper/312:33:263
788999210,20cyclictest0-21swapper/311:00:023
788999210,20cyclictest0-21swapper/310:28:273
788999210,20cyclictest0-21swapper/309:53:263
788999210,20cyclictest0-21swapper/307:53:273
788999210,20cyclictest0-21swapper/307:53:273
788999210,1cyclictest0-21swapper/309:05:003
788999210,0cyclictest22153-21apt-get08:48:253
788999210,0cyclictest0-21swapper/307:33:253
7886992121,0cyclictest31748-21apt-get09:08:252
7886992121,0cyclictest0-21swapper/210:48:242
7886992121,0cyclictest0-21swapper/210:28:272
7886992121,0cyclictest0-21swapper/207:33:262
7886992120,0cyclictest28067-21apt-get11:33:272
7886992120,0cyclictest0-21swapper/209:23:292
7886992119,1cyclictest22108-21munin-run11:15:012
788699210,21cyclictest0-21swapper/212:28:272
788699210,20cyclictest30508-21apt-get07:58:232
788699210,20cyclictest17348-21apt-get08:38:262
788699210,20cyclictest0-21swapper/210:53:272
788699210,20cyclictest0-21swapper/210:13:282
788699210,1cyclictest0-21swapper/212:25:002
788699210,0cyclictest0-21swapper/208:48:262
7882992121,0cyclictest0-21swapper/109:13:261
7882992120,1cyclictest7693-21apt-get08:18:261
7882992120,1cyclictest0-21swapper/108:45:011
7882992120,0cyclictest0-21swapper/109:48:271
7882992120,0cyclictest0-21swapper/109:08:261
7882992118,2cyclictest4792-21perf08:10:011
788299210,20cyclictest0-21swapper/112:23:221
788299210,20cyclictest0-21swapper/109:43:211
788299210,1cyclictest0-21swapper/108:05:011
788299210,0cyclictest18416-21apt-get07:33:261
788299210,0cyclictest0-21swapper/110:48:231
7879992121,0cyclictest0-21swapper/009:33:270
7879992120,1cyclictest3503-21apt-get11:53:290
7879992120,0cyclictest8370-21apt-get07:13:250
7879992118,2cyclictest21621-21cron08:45:000
787999210,21cyclictest9142-21apt-get10:48:240
787999210,20cyclictest0-21swapper/007:28:270
787999210,1cyclictest11938-21apt-get10:53:280
787999210,1cyclictest0-21swapper/012:13:280
787999210,0cyclictest31748-21apt-get09:08:260
7889992020,0cyclictest11742-21apt-get09:33:283
7889992020,0cyclictest10117-21apt-get08:23:293
7889992020,0cyclictest0-21swapper/310:23:213
7889992020,0cyclictest0-21swapper/309:50:013
7889992020,0cyclictest0-21swapper/308:53:263
7889992020,0cyclictest0-21swapper/307:43:293
7889992020,0cyclictest0-21swapper/307:28:263
7889992019,1cyclictest14286-21dpkg09:35:023
788999201,1cyclictest6011-21apt-get09:20:013
788999200,1cyclictest9547-21cron08:19:593
788999200,0cyclictest0-21swapper/309:13:273
788999200,0cyclictest0-21swapper/308:33:283
788999200,0cyclictest0-21swapper/307:38:283
7886992020,0cyclictest20812-21apt-get07:38:282
7886992020,0cyclictest20418-21apt-get09:53:252
7886992020,0cyclictest0-21swapper/212:33:262
7886992020,0cyclictest0-21swapper/212:08:272
7886992020,0cyclictest0-21swapper/208:33:282
7886992019,0cyclictest0-21swapper/210:33:282
7886992018,1cyclictest30611-21sendmail-msp11:40:002
7882992020,0cyclictest0-21swapper/110:50:011
7882992020,0cyclictest0-21swapper/110:28:271
7882992020,0cyclictest0-21swapper/109:35:021
7882992020,0cyclictest0-21swapper/109:23:291
7882992019,0cyclictest0-21swapper/107:23:241
7882992018,1cyclictest0-21swapper/109:33:281
7879992020,0cyclictest2907-21apt-get08:08:250
7879992020,0cyclictest0-21swapper/012:03:210
7879992020,0cyclictest0-21swapper/011:43:210
7879992020,0cyclictest0-21swapper/009:20:020
7879992019,1cyclictest12508-21apt-get08:28:250
7879992018,1cyclictest31307-21apt-get10:23:220
7879992018,1cyclictest14529-21perf11:00:000
7879992018,1cyclictest0-21swapper/009:40:010
7879992018,1cyclictest0-21swapper/007:25:000
787999200,19cyclictest22946-21turbostat.cron10:00:000
787999200,0cyclictest0-21swapper/008:13:270
7889991919,0cyclictest0-21swapper/312:08:283
7889991919,0cyclictest0-21swapper/310:08:273
7889991918,1cyclictest11215-21latency_hist09:29:593
788999190,0cyclictest0-21swapper/311:53:293
7886991919,0cyclictest24549-21apt-get08:53:262
7886991919,0cyclictest0-21swapper/209:33:282
7886991919,0cyclictest0-21swapper/207:13:262
7886991918,1cyclictest0-21swapper/209:18:252
7882991919,0cyclictest9341-21apt-get09:28:261
7882991919,0cyclictest25867-21apt-get10:08:291
7882991919,0cyclictest0-21swapper/112:03:221
7882991919,0cyclictest0-21swapper/110:03:221
7882991918,1cyclictest0-21swapper/108:38:261
7882991917,1cyclictest22179-21munin-plugin-st12:40:001
7879991919,0cyclictest467-21apt-get08:03:290
7879991919,0cyclictest0-21swapper/008:33:270
7879991919,0cyclictest0-21swapper/007:58:250
7879991917,1cyclictest0-21swapper/010:20:000
787999190,1cyclictest17831-21apt-get11:08:260
7889991818,0cyclictest0-21swapper/312:28:273
7889991818,0cyclictest0-21swapper/311:08:253
7889991818,0cyclictest0-21swapper/307:48:283
7889991818,0cyclictest0-21swapper/307:15:023
7889991817,0cyclictest6738-21apt-get10:43:213
788999180,17cyclictest15484-21cron07:25:003
7886991818,0cyclictest13576-21apt-get07:23:242
7886991818,0cyclictest0-21swapper/211:48:252
7886991818,0cyclictest0-21swapper/209:59:592
7886991818,0cyclictest0-21swapper/207:43:292
7882991818,0cyclictest0-21swapper/112:13:271
7882991818,0cyclictest0-21swapper/107:28:271
7882991816,1cyclictest31214-21cron09:04:591
7879991818,0cyclictest0-21swapper/011:33:270
7879991818,0cyclictest0-21swapper/010:13:270
7879991818,0cyclictest0-21swapper/009:48:280
7879991818,0cyclictest0-21swapper/009:13:270
7879991817,1cyclictest27610-21perf07:50:000
7879991817,1cyclictest0-21swapper/012:23:210
7879991816,1cyclictest19252-21sendmail-msp08:39:590
7879991815,2cyclictest25318-21cron10:05:000
7889991717,0cyclictest0-21swapper/311:33:283
7889991717,0cyclictest0-21swapper/311:28:253
7889991717,0cyclictest0-21swapper/310:48:243
7889991714,2cyclictest7179-21idleruntime-cro08:14:593
7889991714,2cyclictest25340-21latency_hist10:05:003
788999170,17cyclictest0-21swapper/312:13:263
7886991717,0cyclictest0-21swapper/209:43:222
7882991717,0cyclictest0-21swapper/111:33:281
7882991717,0cyclictest0-21swapper/110:23:221
7882991717,0cyclictest0-21swapper/108:53:251
7882991717,0cyclictest0-21swapper/108:33:281
7882991716,1cyclictest0-21swapper/111:28:251
7882991715,1cyclictest0-21swapper/108:28:261
7879991717,0cyclictest0-21swapper/010:08:320
7879991717,0cyclictest0-21swapper/007:43:280
7879991716,0cyclictest0-21swapper/012:33:250
7879991711,3cyclictest0-21swapper/010:43:210
787999170,16cyclictest0-21swapper/010:33:340
7889991616,0cyclictest0-21swapper/311:00:003
788999160,15cyclictest8009-21cat12:03:303
788999160,15cyclictest0-21swapper/310:53:323
7886991616,0cyclictest16025-21apt-get07:28:272
7886991616,0cyclictest0-21swapper/209:28:262
7886991615,0cyclictest0-21swapper/208:43:292
7886991614,1cyclictest0-21swapper/212:00:002
7886991613,2cyclictest22769-21munin-run11:20:012
788699160,2cyclictest12930-21diskstats08:28:282
7882991616,0cyclictest0-21swapper/107:58:251
7882991614,1cyclictest5379-21wc11:54:591
7882991614,1cyclictest0-21swapper/112:30:001
7882991614,1cyclictest0-21swapper/110:00:001
7882991613,2cyclictest6049-21idleruntime-cro12:00:001
788299160,2cyclictest18000-21cpuspeed_turbos11:08:271
788299160,15cyclictest0-21swapper/112:20:001
787999160,15cyclictest30022-21sed07:55:010
787999160,15cyclictest30022-21sed07:55:010
787999160,15cyclictest22133-21cron12:39:590
787999160,15cyclictest0-21swapper/012:10:010
7889991515,0cyclictest0-21swapper/312:23:213
7889991515,0cyclictest0-21swapper/311:43:223
7889991515,0cyclictest0-21swapper/308:58:283
7889991514,1cyclictest2332-21taskset08:04:383
788999150,14cyclictest0-21swapper/309:10:003
788699152,13cyclictest766-21snmpd12:12:422
7886991515,0cyclictest31115-21apt-get11:43:222
7886991515,0cyclictest0-21swapper/208:08:252
7886991514,0cyclictest28115-21apt-get07:53:252
7886991514,0cyclictest28115-21apt-get07:53:252
7886991513,1cyclictest0-21swapper/211:30:002
7886991513,1cyclictest0-21swapper/211:23:262
788699150,1cyclictest2538-21irqstats09:13:292
788699150,14cyclictest1686-21apache208:22:412
787999150,14cyclictest0-21swapper/008:49:350
7889991414,0cyclictest0-21swapper/311:24:393
7889991414,0cyclictest0-21swapper/311:13:063
7889991412,1cyclictest3955-21ntp_states08:08:303
788999140,14cyclictest0-21swapper/312:36:273
788999140,14cyclictest0-21swapper/312:15:523
788999140,14cyclictest0-21swapper/311:56:543
788999140,14cyclictest0-21swapper/311:18:323
788999140,14cyclictest0-21swapper/310:10:323
788999140,14cyclictest0-21swapper/309:56:143
788999140,0cyclictest0-21swapper/310:16:343
788999140,0cyclictest0-21swapper/307:11:273
7886991414,0cyclictest0-21swapper/212:35:132
7886991414,0cyclictest0-21swapper/212:15:022
7886991414,0cyclictest0-21swapper/210:08:272
788699140,14cyclictest0-21swapper/211:51:572
788699140,14cyclictest0-21swapper/210:37:172
788699140,14cyclictest0-21swapper/210:17:532
788699140,14cyclictest0-21swapper/209:02:302
788699140,13cyclictest13614-21apt-get09:35:022
788699140,0cyclictest0-21swapper/210:58:582
7882991414,0cyclictest0-21swapper/112:32:151
7882991414,0cyclictest0-21swapper/111:20:491
7882991414,0cyclictest0-21swapper/110:42:221
788299140,14cyclictest0-21swapper/111:17:081
788299140,14cyclictest0-21swapper/110:55:321
788299140,14cyclictest0-21swapper/110:31:281
788299140,14cyclictest0-21swapper/110:18:401
788299140,14cyclictest0-21swapper/108:12:431
788299140,14cyclictest0-21swapper/107:41:081
788299140,0cyclictest0-21swapper/110:35:091
788299140,0cyclictest0-21swapper/108:56:361
788299140,0cyclictest0-21swapper/107:47:231
788299140,0cyclictest0-21swapper/107:11:531
7879991414,0cyclictest0-21swapper/011:23:210
7879991414,0cyclictest0-21swapper/009:50:410
7879991414,0cyclictest0-21swapper/009:35:360
7879991414,0cyclictest0-21swapper/008:53:350
787999140,14cyclictest0-21swapper/012:15:590
787999140,14cyclictest0-21swapper/011:34:590
787999140,14cyclictest0-21swapper/009:16:120
787999140,0cyclictest0-21swapper/011:19:590
787999140,0cyclictest0-21swapper/010:35:140
787999140,0cyclictest0-21swapper/009:25:420
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional