You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 15:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Dec 05, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8350210187,9sleep30-21swapper/307:06:333
849229683,9sleep10-21swapper/107:08:231
833728366,13sleep20-21swapper/207:06:232
860328272,6sleep00-21swapper/007:09:140
874599242,21cyclictest0-21swapper/011:40:120
875599234,13cyclictest0-21swapper/310:00:163
875299220,2cyclictest0-21swapper/211:40:102
875299220,20cyclictest0-21swapper/211:47:232
875299210,21cyclictest0-21swapper/211:56:442
875299210,0cyclictest0-21swapper/210:34:312
8752992018,1cyclictest0-21swapper/210:12:392
875299190,19cyclictest24393-21kworker/2:210:29:592
875299190,18cyclictest0-21swapper/212:00:282
874599197,6cyclictest25583-21perf12:04:590
8745991916,1cyclictest0-21swapper/010:45:140
875299180,1cyclictest0-21swapper/209:41:352
874599185,7cyclictest20270-21sendmail-msp10:40:000
8745991815,1cyclictest0-21swapper/010:43:460
875299170,2cyclictest2605-21cstates12:35:132
8745991715,1cyclictest0-21swapper/007:55:110
8755991616,0cyclictest0-21swapper/310:11:133
875599160,15cyclictest0-21swapper/310:58:533
875299162,1cyclictest0-21swapper/211:53:392
8752991615,1cyclictest5012-21kworker/2:009:10:192
875299160,16cyclictest0-21swapper/212:12:052
875299160,16cyclictest0-21swapper/210:43:082
875299160,15cyclictest22160-21ssh11:35:192
875299160,10cyclictest0-21swapper/211:31:042
875099164,7cyclictest2256-21sh11:20:001
8750991616,0cyclictest0-21swapper/109:43:071
105902160,1sleep10-21swapper/110:58:071
875599150,2cyclictest0-21swapper/312:13:193
875599150,1cyclictest0-21swapper/309:40:113
8752991514,1cyclictest28242-21kworker/2:209:55:192
8752991512,2cyclictest37850irq/131-snd_hda10:50:012
875299150,3cyclictest0-21swapper/209:18:032
875299150,15cyclictest0-21swapper/209:30:322
875299150,14cyclictest0-21swapper/211:25:512
8750991514,1cyclictest0-21swapper/109:55:151
875099150,15cyclictest0-21swapper/110:32:121
8745991514,1cyclictest0-21swapper/011:15:440
8755991410,3cyclictest2299-21cat11:20:003
8755991410,3cyclictest14158-21cron11:30:013
875299141,12cyclictest20240-21munin-run10:39:592
8752991410,3cyclictest23553-21apt08:10:012
8752991410,3cyclictest15382-21wc12:20:262
875299140,1cyclictest0-21swapper/212:17:162
875299140,14cyclictest0-21swapper/209:22:142
875099145,2cyclictest141rcu_preempt12:22:301
875099140,13cyclictest0-21swapper/107:10:121
875099140,12cyclictest8328-21cat10:30:001
874599141,6cyclictest19230-21munin-run07:50:000
874599140,1cyclictest0-21swapper/009:49:390
875599139,3cyclictest31114-21grep09:30:183
875599139,3cyclictest14188-21sed09:15:253
875599133,3cyclictest224-21systemd-journal09:50:003
8755991312,1cyclictest0-21swapper/309:55:583
8755991310,2cyclictest8316-21grep12:15:263
8755991310,2cyclictest7945-21cron11:25:003
8755991310,2cyclictest29483-21sed10:20:183
8755991310,2cyclictest24868-21cat08:10:193
8755991310,2cyclictest21001-21grep12:25:163
8755991310,2cyclictest14519-21tail07:25:283
8752991310,2cyclictest23800-21tr09:50:142
8752991310,2cyclictest23800-21tr09:50:132
8752991310,2cyclictest23710-21grep11:10:162
8752991310,2cyclictest11219-21wc07:15:002
875299130,2cyclictest0-21swapper/210:58:012
875299130,12cyclictest0-21swapper/210:46:342
875299130,12cyclictest0-21swapper/210:06:522
875099139,3cyclictest4585-21grep11:20:271
875099139,3cyclictest2520-21tr08:55:121
875099139,3cyclictest16502-21sed10:10:151
875099139,2cyclictest17249-21grep07:40:151
875099139,2cyclictest14357-21cat11:30:001
8750991311,1cyclictest8436-21wc12:40:011
8750991310,2cyclictest4142-21sort09:35:011
8750991310,2cyclictest26169-21wc09:25:231
8750991310,2cyclictest24337-21cron09:25:011
8750991310,2cyclictest23168-21ssh09:50:001
8750991310,2cyclictest21650-21tail12:25:191
875099130,8cyclictest15720-21missed_timers10:35:201
875099130,1cyclictest0-21swapper/112:01:541
875099130,12cyclictest8134-21wc12:15:271
875099130,0cyclictest0-21swapper/109:19:021
8745991312,1cyclictest0-21swapper/009:21:250
8745991310,2cyclictest32760-21grep12:10:180
8745991310,2cyclictest13235-21grep11:00:180
874599130,12cyclictest25461-21grep08:10:260
875599129,2cyclictest8612-21wget10:30:163
875599129,2cyclictest29451-21grep11:40:263
875599129,2cyclictest28752-21cron11:15:003
875599129,2cyclictest27970-21sort10:45:183
875599129,2cyclictest26282-21tr12:05:173
875599129,2cyclictest25979-21head09:25:213
875599129,2cyclictest24683-21cat09:50:183
875599129,2cyclictest24683-21cat09:50:183
875599129,2cyclictest19132-21grep09:20:153
875599125,6cyclictest27180-21sendmail11:40:013
875599121,1cyclictest0-21swapper/311:00:133
875599121,10cyclictest0-21swapper/311:49:173
875599120,11cyclictest28090-21tr12:30:163
875299129,2cyclictest30853-21basename08:39:592
875299129,2cyclictest28634-21grep12:30:192
875299129,2cyclictest25601-21grep09:25:182
875299129,2cyclictest23601-21tr10:15:222
875299129,2cyclictest2220-21wc08:55:012
875299129,2cyclictest22174-21grep12:25:242
875299128,3cyclictest18193-21cut09:45:162
875299128,3cyclictest155122sleep207:34:162
875299127,4cyclictest19944-21grep11:05:272
8752991211,1cyclictest5968-21kworker/2:209:35:382
8752991211,1cyclictest0-21swapper/210:03:472
875299120,12cyclictest0-21swapper/212:09:012
875099129,2cyclictest7990-21awk11:50:011
875099129,2cyclictest4334-21grep10:00:211
875099129,2cyclictest31724-21sort08:40:161
875099129,2cyclictest29328-21cron09:55:011
875099129,2cyclictest29328-21cron09:55:001
875099129,2cyclictest26764-21grep12:05:201
875099129,2cyclictest22512-21grep10:15:151
875099129,2cyclictest21808-21grep07:55:251
875099129,2cyclictest10583-21dump-pmu-power09:40:001
875099128,2cyclictest26642-21wc08:20:011
875099121,5cyclictest224-21systemd-journal07:37:301
875099121,1cyclictest141rcu_preempt11:54:031
875099120,11cyclictest29175-21grep10:45:281
875099120,11cyclictest20026-21head11:05:281
874599129,2cyclictest6806-21grep09:10:130
874599129,2cyclictest3035-21grep12:35:140
874599129,2cyclictest26301-21ssh11:39:030
874599129,2cyclictest25136-21cat09:50:250
874599129,2cyclictest25136-21cat09:50:240
874599129,2cyclictest23543-21cron08:10:000
874599129,2cyclictest19943-21cat12:00:000
874599129,2cyclictest16712-21cut07:40:130
874599129,2cyclictest1218-21tr10:50:150
874599126,5cyclictest21614-21apache210:05:120
874599125,3cyclictest3992-21unixbench-2d10:25:280
874599122,8cyclictest4257-21sensors11:20:250
874599122,6cyclictest18125-21apache_accesses11:05:120
8745991211,1cyclictest0-21swapper/012:25:050
874599120,11cyclictest26850-21rm10:18:270
875599118,2cyclictest8964-21tr11:50:163
875599118,2cyclictest4704-21egrep09:35:133
875599118,2cyclictest4539-21rm10:52:273
875599118,2cyclictest3275-21wc10:25:203
875599118,2cyclictest30182-21cron08:35:003
875599118,2cyclictest2467-21tr12:35:133
875599118,2cyclictest24579-21ssh10:15:513
875599118,2cyclictest21385-21chrt10:40:143
875599118,2cyclictest21302-21uname12:00:163
875599118,2cyclictest18456-21ssh11:32:423
875599118,2cyclictest15982-21chrt07:37:223
875599118,2cyclictest15957-21tr10:35:223
875599118,2cyclictest15355-21uname11:55:223
875599118,2cyclictest14028-21grep12:20:143
875599118,2cyclictest11301-21grep10:05:223
875599116,2cyclictest224-21systemd-journal08:30:003
875599114,4cyclictest6016-21/usr/sbin/munin09:10:163
875599113,2cyclictest224-21systemd-journal07:43:353
875299118,3cyclictest29276-21chrt08:26:252
875299118,2cyclictest3699-21ssh11:20:192
875299118,2cyclictest32707-21grep08:40:262
875299118,2cyclictest29977-21head11:15:182
875299118,2cyclictest25468-21tail08:10:262
875299118,2cyclictest22650-21seq08:01:432
875299118,2cyclictest20142-21grep07:55:132
875299118,2cyclictest18419-21taskset07:42:522
875299118,2cyclictest15116-21sh11:01:202
875299114,3cyclictest224-21systemd-journal08:25:002
8752991110,1cyclictest24393-21kworker/2:210:23:382
875099118,2cyclictest32561-21sh12:33:221
875099118,2cyclictest31410-21ssh10:21:281
875099118,2cyclictest31098-21taskset11:42:171
875099118,2cyclictest30682-21taskset08:38:291
875099118,2cyclictest25423-21sed08:10:251
875099118,2cyclictest25365-21ssh11:38:051
875099118,2cyclictest19430-21cat07:50:011
875099118,2cyclictest18748-21ssh11:33:001
875099118,2cyclictest1529-21sed10:50:171
875099118,2cyclictest12378-21ssh10:06:051
875099117,2cyclictest6346-21tr09:10:121
875099110,10cyclictest12611-21grep11:00:151
874599119,1cyclictest8841-21grep10:30:160
874599119,1cyclictest11298-21tr09:40:140
874599118,2cyclictest9282-21cat07:10:130
874599118,2cyclictest8882-21expr11:25:150
874599118,2cyclictest7844-21ssh11:49:540
874599118,2cyclictest4700-21ssh09:35:130
874599118,2cyclictest30863-21cat08:40:000
874599118,2cyclictest308-21ssh09:31:020
874599118,2cyclictest30111-21sshd08:34:080
874599118,2cyclictest29725-21chrt09:55:110
874599118,2cyclictest28445-21grep12:30:180
874599118,2cyclictest27191-21sh08:25:000
874599118,2cyclictest25911-21ssh12:05:100
874599118,2cyclictest25753-21ssh11:10:580
874599118,2cyclictest25070-21tr09:25:160
874599118,2cyclictest19573-21ssh10:12:300
874599118,2cyclictest17461-21ssh12:22:050
874599118,2cyclictest16386-21ssh09:18:190
874599118,2cyclictest10242-21wc11:50:260
874599114,1cyclictest9344-21munin-run10:05:000
874599112,8cyclictest1348-21munin-node09:04:420
875599108,1cyclictest31243-21grep08:40:153
875599108,1cyclictest30463-21chrt08:35:393
875599108,1cyclictest26361-21chrt08:17:443
875599108,1cyclictest19804-21taskset07:54:373
875599108,1cyclictest18937-21seq07:46:333
875599108,1cyclictest1875-21taskset08:53:103
875599108,1cyclictest18025-21ssh11:05:083
875599108,1cyclictest12273-21chrt07:21:153
875599108,1cyclictest115412sleep307:15:113
875599108,1cyclictest11022sleep308:46:213
875599107,2cyclictest3011-21head08:55:143
875599107,2cyclictest15189-21seq07:30:093
875599100,9cyclictest5355-21taskset09:06:043
875299108,1cyclictest5166-21sort09:05:012
875299108,1cyclictest29812-21chrt08:30:172
875299108,1cyclictest19752-21chrt07:53:592
875299108,1cyclictest19089-21taskset07:48:302
875299108,1cyclictest123782sleep207:22:342
875299107,2cyclictest14067-21head07:25:232
875299107,2cyclictest11890-21taskset07:19:332
875299100,9cyclictest4391-21chrt08:59:542
875099108,1cyclictest28951-21sed08:25:251
875099108,1cyclictest270762sleep108:23:391
875099108,1cyclictest23899-21ls11:10:161
875099108,1cyclictest197632sleep107:54:061
875099108,1cyclictest1846-21taskset08:52:461
875099108,1cyclictest14139-21ssh11:55:041
875099108,1cyclictest13779-21grep07:25:181
875099107,2cyclictest726-21sed12:10:211
875099107,2cyclictest5425-21taskset09:06:581
875099107,2cyclictest20776-21ssh10:40:061
875099107,2cyclictest11825-21taskset07:18:431
875099102,4cyclictest224-21systemd-journal09:03:511
875099102,4cyclictest224-21systemd-journal08:34:591
875099101,5cyclictest224-21systemd-journal07:35:001
874599109,1cyclictest0-21swapper/008:55:120
874599108,1cyclictest6873-21rm10:55:080
874599108,1cyclictest6398-21ssh12:15:110
874599108,1cyclictest5757-21sed09:10:000
874599108,1cyclictest28617-21ssh10:20:100
874599108,1cyclictest27707-21cut08:25:140
874599108,1cyclictest16737-21ssh11:30:450
874599108,1cyclictest142382sleep007:25:240
874599108,1cyclictest1288-21chrt08:48:440
874599108,1cyclictest122452sleep007:20:530
874599108,1cyclictest118252sleep007:18:430
874599107,2cyclictest31874-21grep08:40:170
874599107,2cyclictest16207-21sh07:40:000
874599107,2cyclictest15332-21taskset07:31:580
87559997,1cyclictest9670-21tune2fs07:10:143
87559997,1cyclictest4827-21taskset09:02:193
87559997,1cyclictest26977-21taskset08:22:243
87559997,1cyclictest233532sleep308:07:443
87559997,1cyclictest20686-21diskstats07:55:143
87559996,2cyclictest22530-21taskset08:00:093
87529997,1cyclictest262072sleep208:15:482
87529997,1cyclictest11482sleep208:46:562
87509997,1cyclictest22548-21chrt08:00:231
87509997,1cyclictest123652sleep107:22:251
87509994,2cyclictest224-21systemd-journal08:49:591
87459997,1cyclictest26259-21sshd08:16:290
87459997,1cyclictest225592sleep008:00:300
87459997,1cyclictest196012sleep007:52:050
87529986,1cyclictest5494-21chrt09:07:522
87529983,2cyclictest224-21systemd-journal07:35:302
87509986,1cyclictest234062sleep108:08:261
87459983,2cyclictest224-21systemd-journal08:53:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional