You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 22:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Nov 25, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32260210175,9sleep10-21swapper/107:05:211
82829683,9sleep30-21swapper/307:06:123
102228666,9sleep00-21swapper/007:08:410
3141828164,13sleep20-21swapper/207:05:032
121799330,1cyclictest0-21swapper/309:54:483
121099260,6cyclictest26545-21ls11:50:201
121099253,3cyclictest32621-21/usr/sbin/munin12:20:211
121099221,2cyclictest26219-21diskmemload10:18:291
121099211,5cyclictest1320-21wc09:15:251
121099210,1cyclictest0-21swapper/110:05:501
121099210,0cyclictest0-21swapper/109:45:141
120699210,2cyclictest0-21swapper/010:31:350
208732200,0sleep00-21swapper/012:10:350
121799204,11cyclictest0-21swapper/308:20:153
121099202,4cyclictest29205-21expr09:40:151
121099202,2cyclictest21717-21perl11:20:181
121599190,1cyclictest0-21swapper/208:05:182
121599190,19cyclictest0-21swapper/209:35:072
121599190,19cyclictest0-21swapper/209:35:062
121099195,1cyclictest26219-21diskmemload11:37:541
121099190,5cyclictest10933-21sh07:30:001
121099190,19cyclictest0-21swapper/111:44:181
1215991810,7cyclictest61750irq/125-enp2s0-09:10:282
121599180,18cyclictest0-21swapper/210:48:092
121099187,10cyclictest16119-21sh09:29:441
121099181,1cyclictest0-21swapper/111:25:581
121099180,5cyclictest26219-21diskmemload11:46:531
121099180,3cyclictest0-21swapper/107:35:151
120699180,16cyclictest0-21swapper/008:50:120
121099176,3cyclictest26219-21diskmemload12:27:401
121099174,3cyclictest26219-21diskmemload12:30:161
121099173,1cyclictest3786-21sh11:31:391
121099172,4cyclictest19615-21ssh09:31:181
121099171,4cyclictest5679-21fschecks_count10:40:181
121099171,4cyclictest12853-21fschecks_time08:40:181
121099170,1cyclictest0-21swapper/109:05:141
120699175,6cyclictest20571-21munin-run11:20:000
121599160,15cyclictest0-21swapper/212:19:302
121099169,6cyclictest6967-21head07:20:161
121099168,7cyclictest16977-21cut08:50:131
121099164,11cyclictest0-21swapper/111:10:111
121099163,12cyclictest0-21swapper/109:11:291
121099160,4cyclictest0-21swapper/110:32:211
121099160,2cyclictest0-21swapper/108:25:201
121799159,5cyclictest18082-21munin-run10:25:003
1215991514,1cyclictest0-21swapper/211:10:182
1215991514,1cyclictest0-21swapper/211:09:122
121599150,1cyclictest0-21swapper/212:30:582
121599150,15cyclictest0-21swapper/209:43:402
121099158,6cyclictest21542-21sh12:11:191
121099158,6cyclictest13527-21cat07:35:001
121099158,6cyclictest11868-21gltestperf10:45:181
121099158,6cyclictest1042-21ssh10:35:501
121099157,7cyclictest5075-21cat12:00:001
121099157,7cyclictest10784-21sh12:04:271
121099154,7cyclictest22969-21apt-get09:35:151
121099154,7cyclictest22969-21apt-get09:35:141
121099153,2cyclictest26219-21diskmemload10:25:201
121099152,4cyclictest224-21systemd-journal08:33:591
121099152,12cyclictest0-21swapper/109:00:151
120699151,2cyclictest0-21swapper/009:31:470
1206991510,3cyclictest26219-21diskmemload11:50:380
121799144,4cyclictest15264-21ssh09:55:213
1217991411,2cyclictest905-21grep11:30:133
121799140,8cyclictest7829-21cut09:50:013
121799140,13cyclictest11481-21sh12:28:293
121599144,9cyclictest7428-21munin-node10:22:102
121599143,2cyclictest224-21systemd-journal10:35:002
121599141,13cyclictest26165-21kworker/2:209:16:162
1215991411,2cyclictest20815-21aten2_r7power_p07:50:132
1215991410,3cyclictest23823-21cron09:05:002
121599140,2cyclictest0-21swapper/212:28:192
121099148,5cyclictest20016-21ssh12:35:141
121099148,5cyclictest17694-21grep10:50:161
121099148,5cyclictest17694-21grep10:50:151
121099148,5cyclictest13942-21ssh12:05:391
121099147,6cyclictest8430-21cut09:50:161
121099147,6cyclictest3563-21idleruntime-cro07:15:001
121099147,6cyclictest341-21basename08:15:151
121099147,6cyclictest25566-21sh12:15:091
121099147,6cyclictest16048-21cat11:15:201
121099141,4cyclictest224-21systemd-journal10:00:011
1210991411,2cyclictest23172-21ssh10:55:151
121099140,4cyclictest224-21systemd-journal08:40:001
121099140,1cyclictest0-21swapper/109:24:371
121099140,13cyclictest4332-21ssh11:05:301
121099140,13cyclictest4063-21cut07:15:171
1206991411,2cyclictest20465-21munin-plugin-st07:50:000
1206991410,3cyclictest27462-21ssh11:25:170
120699140,13cyclictest0-21swapper/012:20:170
121799139,3cyclictest25574-21ssh11:50:153
121799139,3cyclictest1727-21sh09:45:003
121799139,3cyclictest13855-21tr07:35:153
121799139,2cyclictest21442-21cat09:00:003
121799139,2cyclictest12753-21wc10:45:253
121799134,2cyclictest14837-21gzip12:30:223
1217991310,2cyclictest7671-21grep08:30:143
1217991310,2cyclictest32456-21sed11:55:193
1217991310,2cyclictest16220-21tr07:40:153
1217991310,2cyclictest16220-21tr07:40:153
1217991310,2cyclictest12025-21sort07:30:193
121799130,8cyclictest0-21swapper/312:00:073
121599138,4cyclictest3889-21ssh11:58:452
121599137,2cyclictest224-21systemd-journal07:20:012
1215991311,1cyclictest32-21ksoftirqd/210:06:162
1215991311,1cyclictest22744-21grep11:20:282
121599131,0cyclictest141rcu_preempt07:10:162
1215991310,2cyclictest8616-21cat11:35:262
1215991310,2cyclictest7642-21sh09:50:012
1215991310,2cyclictest27111-21grep08:00:292
1215991310,2cyclictest26009-21rm11:50:162
1215991310,2cyclictest25761-21grep09:05:262
121599130,12cyclictest4340-21tail08:20:282
121599130,12cyclictest26471-21basename12:40:012
121599130,0cyclictest0-21swapper/211:30:562
121099138,4cyclictest213942sleep108:59:411
121099137,5cyclictest26294-21sh08:00:201
121099137,5cyclictest22076-21cut07:50:211
121099137,5cyclictest14910-21chrt10:21:271
121099136,6cyclictest19841-21ssh10:00:141
121099136,1cyclictest703-21sh10:10:231
121099132,2cyclictest224-21systemd-journal11:00:011
121099131,11cyclictest0-21swapper/107:40:141
121099131,11cyclictest0-21swapper/107:40:131
121099130,12cyclictest24362-21wc07:55:231
1206991312,1cyclictest0-21swapper/010:40:140
1206991311,1cyclictest235522sleep009:01:370
1206991310,2cyclictest697-21sed10:35:270
1206991310,2cyclictest5338-21cut09:20:140
1206991310,2cyclictest14260-21cut07:35:180
1206991310,2cyclictest14243-21head11:40:180
1206991310,2cyclictest13857-21grep08:40:280
120699130,8cyclictest0-21swapper/012:15:140
120699130,4cyclictest0-21swapper/009:25:150
120699130,2cyclictest0-21swapper/009:39:200
120699130,2cyclictest0-21swapper/009:39:190
120699130,1cyclictest0-21swapper/009:42:280
121799129,2cyclictest9008-21grep11:10:153
121799129,2cyclictest8419-21sed11:10:003
121799129,2cyclictest4549-21cron10:40:003
121799129,2cyclictest26658-21head10:05:183
121799129,2cyclictest21970-21tr09:00:183
121799129,2cyclictest16831-21cat08:50:003
121799129,2cyclictest12696-21tail12:05:163
121799129,2cyclictest10057-21grep07:25:233
121799128,3cyclictest8490-21wc11:35:263
121799128,3cyclictest545-21cut12:20:183
121799128,3cyclictest338-21grep09:15:173
121799127,2cyclictest19282-21ssh09:30:573
121799121,2cyclictest31871-21ssh11:28:583
1217991211,1cyclictest0-21swapper/308:41:373
121599129,3cyclictest0-21swapper/211:15:132
121599129,2cyclictest9535-21latency_hist08:34:592
121599129,2cyclictest8328-21cut09:50:152
121599129,2cyclictest31986-21sed08:10:262
121599129,2cyclictest18951-21sort07:45:152
121599129,2cyclictest18761-21grep09:30:272
121599129,2cyclictest18643-21basename10:25:142
121599128,3cyclictest16188-21rm12:07:542
121599128,3cyclictest15733-21ntp_states08:45:202
121599127,4cyclictest6225-21ssh10:40:242
121099128,3cyclictest18897-21sed07:45:141
121099125,2cyclictest29801-21taskset08:07:011
120699129,2cyclictest8209-21cat11:10:000
120699129,2cyclictest8134-21grep08:30:170
120699129,2cyclictest7173-21grep11:35:150
120699129,2cyclictest5154-21grep08:25:150
120699129,2cyclictest26431-21ssh11:24:580
120699129,2cyclictest23527-21expr10:55:160
120699129,2cyclictest20345-21grep11:45:170
120699129,2cyclictest19352-21cron10:00:000
120699129,2cyclictest13675-21grep12:30:160
120699128,3cyclictest32116-21grep10:10:150
120699128,3cyclictest31455-21cron10:10:000
120699128,2cyclictest26730-21grep09:10:140
120699120,3cyclictest0-21swapper/012:01:500
121799119,1cyclictest21442-21grep12:35:243
121799118,2cyclictest6248-21rm10:15:183
121799118,2cyclictest5073-21grep10:40:153
121799118,2cyclictest39-21ksoftirqd/310:26:113
121799118,2cyclictest31788-21ssh11:01:413
121799118,2cyclictest31044-21rm09:13:533
121799118,2cyclictest29767-21ssh10:33:563
121799118,2cyclictest28144-21grep08:05:163
121799118,2cyclictest27366-21taskset08:02:033
121799118,2cyclictest26968-21open_inodes12:15:223
121799118,2cyclictest26730-21ssh10:57:473
121799118,2cyclictest25456-21sendmail_mailtr09:05:233
121799118,2cyclictest23220-21ssh12:12:593
121799118,2cyclictest23206-21grep07:55:163
121799118,2cyclictest22967-21rm10:02:043
121799118,2cyclictest22869-21ssh09:35:103
121799118,2cyclictest22869-21ssh09:35:103
121799118,2cyclictest20057-21expr11:45:163
121799118,2cyclictest19957-21grep07:45:263
121799118,2cyclictest16899-21ssh11:15:353
121799118,2cyclictest15567-21chrt11:40:433
121799118,2cyclictest1487-21cut08:15:213
121799118,2cyclictest11646-21sed09:25:183
121799117,3cyclictest31343-21wc08:10:193
121799117,3cyclictest21125-21grep11:20:153
121799116,3cyclictest0-21swapper/307:50:113
121599119,1cyclictest24423-21cut10:55:232
121599119,1cyclictest17331-21cat08:50:162
121599119,1cyclictest10543-21head08:35:182
121599118,2cyclictest6727-21sendmail_mailtr09:20:252
121599118,2cyclictest6095-21grep12:00:192
121599118,2cyclictest29152-21ssh11:25:472
121599118,2cyclictest28999-21tr11:00:152
121599118,2cyclictest24430-21ssh12:14:162
121599118,2cyclictest2135-21ssh10:11:282
121599118,2cyclictest20912-21grep11:45:362
121599118,2cyclictest20088-21cut08:55:172
121599118,2cyclictest18022-21sh11:43:482
121599118,2cyclictest15196-21ssh09:28:362
121599118,2cyclictest12950-21sshd07:30:282
121599117,4cyclictest32512-21ntpq10:35:212
121599117,3cyclictest18505-21grep10:50:222
121599117,3cyclictest18505-21grep10:50:222
121599117,3cyclictest1433-21cat12:20:232
121599116,4cyclictest20518-21cut10:00:182
121599113,7cyclictest6404-21cut10:15:202
121099118,2cyclictest15285-21head08:45:161
121099114,2cyclictest3447-21sed08:20:191
121099110,4cyclictest30590-21cpuspeed_turbos08:10:141
120699118,2cyclictest8762-21grep12:25:370
120699118,2cyclictest7518-21sed10:15:280
120699118,2cyclictest7248-21ssh09:49:300
120699118,2cyclictest6203-21cat07:20:110
120699118,2cyclictest5940-21cron07:20:000
120699118,2cyclictest3698-21sshd11:58:300
120699118,2cyclictest29248-21expr11:00:160
120699118,2cyclictest24743-21grep09:05:180
120699118,2cyclictest23137-21grep07:55:150
120699118,2cyclictest22335-21grep07:50:250
120699118,2cyclictest20090-21wget12:35:140
120699118,2cyclictest18744-21sort10:25:130
120699118,2cyclictest14324-21ssh11:14:500
120699118,2cyclictest14001-21ssh12:05:430
120699118,2cyclictest13173-21ssh10:45:470
120699118,2cyclictest11484-21tr07:30:160
120699118,2cyclictest106802sleep007:26:520
120699117,3cyclictest12290-21tr10:20:150
120699110,3cyclictest0-21swapper/010:51:250
120699110,3cyclictest0-21swapper/010:51:240
121799108,1cyclictest6332-21cut07:20:133
121799108,1cyclictest119042sleep308:39:593
121799107,2cyclictest6176-21chrt08:25:213
121799107,2cyclictest2483-21ssh10:11:513
121799107,2cyclictest17924-21cat08:50:213
121799107,2cyclictest16974-21ssh10:50:033
121799107,2cyclictest16974-21ssh10:50:033
121799105,4cyclictest5210-21/usr/sbin/munin09:20:283
121799100,9cyclictest3436-21chrt07:13:333
121599108,1cyclictest7608-21basename07:20:212
121599108,1cyclictest6579-21cat08:25:262
121599108,1cyclictest16210-21cut07:40:152
121599108,1cyclictest16210-21cut07:40:152
121599108,1cyclictest16023-21rm09:55:322
121599108,1cyclictest14262-21uniq07:35:182
121599107,2cyclictest23523-21expr07:55:172
121599107,2cyclictest1491-21grep08:15:222
121599107,2cyclictest12854-21sort08:40:182
121599107,2cyclictest10040-21pmu-power07:25:232
120699108,1cyclictest29697-21taskset08:05:380
120699108,1cyclictest26150-21fschecks_time08:00:200
120699108,1cyclictest22216-21ssh10:01:060
120699108,1cyclictest214192sleep008:59:570
120699108,1cyclictest14581-21apt-get08:45:130
120699107,2cyclictest3237-21sed08:20:170
120699107,2cyclictest31094-21head08:10:170
120699107,2cyclictest2019-21cut11:30:190
120699107,2cyclictest10530-21tail08:35:190
120699107,2cyclictest10445-21ssh09:50:500
120699106,3cyclictest1787-21wc08:15:250
120699101,7cyclictest2079-21fschecks_count07:10:170
120699101,1cyclictest0-21swapper/007:40:150
120699101,1cyclictest0-21swapper/007:40:140
120699100,9cyclictest32258-21cpu09:15:150
12179997,1cyclictest57432sleep307:17:343
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional