You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 02:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 12, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1427329263,25sleep00-21swapper/007:07:470
1414028470,9sleep20-21swapper/207:06:052
1414128067,9sleep30-21swapper/307:06:063
1417627558,13sleep10-21swapper/107:06:321
1453299321,28cyclictest31948-21/usr/sbin/munin11:20:222
300112310,1sleep10-21swapper/112:16:411
14523993130,1cyclictest0-21swapper/008:13:420
1453299300,29cyclictest0-21swapper/211:34:542
1453299300,17cyclictest0-21swapper/211:50:092
1453299290,28cyclictest0-21swapper/212:10:392
1453299280,27cyclictest0-21swapper/210:44:182
1453299280,18cyclictest0-21swapper/210:16:072
1453299270,27cyclictest0-21swapper/211:40:532
14523992714,9cyclictest7886-21sh12:23:140
1453299260,25cyclictest0-21swapper/210:47:232
1453299250,21cyclictest0-21swapper/210:24:472
1452399254,2cyclictest537-21snmpd09:43:100
1452399250,9cyclictest0-21swapper/011:54:020
1452399250,2cyclictest0-21swapper/009:32:100
1452899233,19cyclictest23912-21sh10:36:061
14523992323,0cyclictest0-21swapper/009:38:100
1452399230,22cyclictest0-21swapper/009:48:060
1453299220,21cyclictest0-21swapper/211:08:372
1453299220,18cyclictest0-21swapper/211:15:272
1452899220,20cyclictest0-21swapper/110:45:581
14528992117,4cyclictest25-21ksoftirqd/109:33:321
1453299200,18cyclictest0-21swapper/211:39:452
1453299200,15cyclictest0-21swapper/211:29:272
1452399200,3cyclictest0-21swapper/010:46:540
1452399200,17cyclictest0-21swapper/012:15:530
1452399200,17cyclictest0-21swapper/011:07:060
1453299190,1cyclictest0-21swapper/211:10:182
14528991918,1cyclictest0-21swapper/110:28:471
1452899190,19cyclictest0-21swapper/111:27:521
1452399198,2cyclictest0-21swapper/010:40:190
1452399194,7cyclictest24161-21munin-run11:34:590
14523991912,6cyclictest31948-21seq12:37:290
1452399190,1cyclictest0-21swapper/012:02:100
14532991817,1cyclictest0-21swapper/212:33:132
14532991817,1cyclictest0-21swapper/210:33:222
14532991816,1cyclictest0-21swapper/212:25:112
14532991816,1cyclictest0-21swapper/210:39:212
1453299180,4cyclictest0-21swapper/209:16:022
1453299180,18cyclictest0-21swapper/212:08:072
14528991816,2cyclictest9648-21kworker/1:012:27:021
1452899181,14cyclictest27181-21aten2_r7power_v10:00:131
1452899180,18cyclictest8610-21kworker/1:210:50:131
1452899180,18cyclictest0-21swapper/111:41:301
1452899180,18cyclictest0-21swapper/109:40:431
1452399189,2cyclictest4738-21dump-pmu-power09:05:000
1452399181,1cyclictest141rcu_preempt12:11:340
1452399181,17cyclictest0-21swapper/010:37:500
14523991810,1cyclictest28424-21grep07:35:250
1453299171,2cyclictest0-21swapper/211:02:152
1453299171,0cyclictest0-21swapper/209:32:072
1453299170,2cyclictest0-21swapper/212:35:512
1453299170,17cyclictest0-21swapper/209:46:292
1453299170,13cyclictest0-21swapper/210:03:132
1452899171,1cyclictest7108-21diskmemload09:22:121
1452899171,15cyclictest18424-21grep07:15:191
14523991711,1cyclictest0-21swapper/011:45:110
1452399170,2cyclictest0-21swapper/012:28:010
1453499160,16cyclictest0-21swapper/310:20:163
1453499160,15cyclictest0-21swapper/309:47:013
1453299161,15cyclictest0-21swapper/210:05:122
1453299160,1cyclictest0-21swapper/208:10:132
1453299160,15cyclictest0-21swapper/209:56:572
1452899160,16cyclictest0-21swapper/109:55:121
1452399169,2cyclictest7647-21head08:00:210
1452399165,6cyclictest12555-21/usr/sbin/munin10:30:100
14523991610,2cyclictest30522-21ls07:40:200
14523991610,1cyclictest5245-21tr09:05:130
1452399160,2cyclictest0-21swapper/012:34:270
14534991515,0cyclictest0-21swapper/311:00:113
14534991514,1cyclictest0-21swapper/312:15:193
1453499150,14cyclictest0-21swapper/310:04:473
14532991514,1cyclictest0-21swapper/210:55:062
1453299150,2cyclictest0-21swapper/211:45:122
1453299150,15cyclictest0-21swapper/212:21:572
1453299150,15cyclictest0-21swapper/210:52:302
1453299150,14cyclictest0-21swapper/209:35:022
14523991513,1cyclictest4368-21ssh10:25:130
1452399150,3cyclictest0-21swapper/010:54:000
14534991412,2cyclictest0-21swapper/312:09:553
14534991412,1cyclictest7108-21diskmemload10:55:133
1453499141,13cyclictest0-21swapper/312:39:583
14534991411,2cyclictest27267-21df07:35:143
14534991410,3cyclictest31963-21sed08:50:263
14534991410,3cyclictest11609-21awk09:50:223
1453499140,14cyclictest0-21swapper/310:33:073
1453299142,1cyclictest55850irq/129-enp2s0-10:10:182
1453299140,14cyclictest0-21swapper/209:23:102
14528991413,1cyclictest0-21swapper/108:52:201
14528991411,3cyclictest38150irq/131-snd_hda09:50:281
14528991410,3cyclictest27460-21grep07:35:161
14528991410,3cyclictest20579-21grep10:34:591
14528991410,3cyclictest12313-21tr08:10:201
14523991413,1cyclictest0-21swapper/011:29:430
14523991413,1cyclictest0-21swapper/011:10:180
14523991413,1cyclictest0-21swapper/009:55:140
1453499139,3cyclictest27916-21grep10:19:593
1453499139,3cyclictest18231-21wc08:25:003
1453499133,9cyclictest5687-21ssh10:25:203
1453499131,11cyclictest0-21swapper/311:29:523
14534991310,2cyclictest405-21head11:20:173
14534991310,2cyclictest24584-21cut07:30:133
14534991310,2cyclictest21801-21cut10:35:163
14534991310,2cyclictest18892-21grep07:15:253
1453499130,12cyclictest8304-21cat11:44:593
1453299130,4cyclictest0-21swapper/209:12:332
1453299130,11cyclictest0-21swapper/209:44:102
1452899139,3cyclictest966-21sed08:55:171
1452899139,3cyclictest30859-21wc10:40:171
1452899138,4cyclictest24336-21ssh12:33:021
1452899138,4cyclictest12627-21ssh11:08:151
1452899134,2cyclictest3829-21wc12:20:231
14528991310,2cyclictest31229-21cut11:00:131
14528991310,2cyclictest28043-21grep08:45:161
14528991310,2cyclictest25379-21sort11:35:161
14528991310,2cyclictest15074-21tr07:10:141
1452899130,12cyclictest23506-21head10:55:151
1452399139,3cyclictest0-21swapper/010:08:480
14523991310,2cyclictest6493-21cron11:05:000
1452399130,8cyclictest16949-21perf07:15:000
85642120,1sleep10-21swapper/109:49:091
1453499129,2cyclictest9638-21sh09:30:133
1453499129,2cyclictest4920-21wc10:05:253
1453499129,2cyclictest31032-21grep07:40:253
1453499129,2cyclictest26652-21grep09:40:153
1453499129,2cyclictest2664-21wc12:20:153
1453499129,2cyclictest26083-21grep11:55:173
1453499129,2cyclictest2330-21grep07:50:163
1453499129,2cyclictest17246-21sh11:10:253
1453499129,2cyclictest12623-21grep12:25:253
1453499129,2cyclictest11511-21apache_accesses10:10:113
1453499128,2cyclictest7848-21grep11:05:163
1453499128,2cyclictest1329-21grep08:55:203
1453499120,11cyclictest6694-21grep08:00:133
1453499120,0cyclictest0-21swapper/309:28:353
1453499120,0cyclictest0-21swapper/309:28:353
1453299129,2cyclictest4629-21tr10:25:142
14532991211,1cyclictest0-21swapper/212:00:562
14532991210,1cyclictest27955-21tr07:35:202
14532991210,1cyclictest24165-21wc08:35:182
1453299120,1cyclictest2941-21ssh09:25:392
1453299120,1cyclictest2941-21ssh09:25:382
1453299120,1cyclictest0-21swapper/211:56:342
1453299120,11cyclictest0-21swapper/212:19:172
1452899129,2cyclictest8725-21wc08:05:001
1452899129,2cyclictest8378-21df09:10:151
1452899129,2cyclictest3493-21ssh10:05:131
1452899129,2cyclictest29206-21grep12:35:241
1452899129,2cyclictest24203-21ssh11:15:141
1452899129,2cyclictest1682-21tr12:00:131
1452899129,2cyclictest15281-21grep08:15:251
1452899129,2cyclictest10708-21grep11:45:251
1452899129,2cyclictest10559-21ssh12:05:171
1452899128,3cyclictest11844-21grep10:10:141
1452899126,3cyclictest24452-21tr07:30:131
14528991211,1cyclictest31602-21kworker/1:110:18:121
14528991211,1cyclictest0-21swapper/111:50:341
1452399129,2cyclictest9089-21grep08:05:140
1452399129,2cyclictest396-21cut11:20:160
1452399129,2cyclictest28422-21grep10:00:200
1452399129,2cyclictest25521-21chrt11:55:120
1452399129,2cyclictest20449-21cat10:15:140
1452399129,2cyclictest15244-21grep08:15:250
1452399129,2cyclictest12143-21grep09:50:260
1452399122,4cyclictest0-21swapper/008:25:140
1452399120,4cyclictest0-21swapper/009:28:430
1452399120,4cyclictest0-21swapper/009:28:420
1452399120,11cyclictest4251-21grep07:55:140
1452399120,11cyclictest2329-21cron09:00:000
229332111,1sleep37108-21diskmemload12:31:563
1453499118,2cyclictest8909-21ssh11:45:123
1453499118,2cyclictest7384-21sh09:10:013
1453499118,2cyclictest32170-21chrt10:40:463
1453499118,2cyclictest3172-21grep09:00:143
1453499118,2cyclictest29025-21ntp_kernel_err08:45:223
1453499118,2cyclictest26835-21ssh11:35:343
1453499118,2cyclictest25880-21ssh11:15:373
1453499118,2cyclictest24532-21python08:35:233
1453499118,2cyclictest24090-21grep07:30:003
1453499118,2cyclictest21484-21ssh12:11:313
1453499118,2cyclictest20707-21uname07:20:193
1453499118,2cyclictest19799-21ssh11:31:353
1453499118,2cyclictest17695-21df_abs11:50:133
1453499118,2cyclictest16433-21aten2_r7power_v09:15:133
1453499118,2cyclictest15458-21sed10:50:193
1453499118,2cyclictest1422-21wc07:50:003
1453499118,2cyclictest12062-21cat08:10:173
1453499118,2cyclictest10160-21ssh10:47:223
1453499118,2cyclictest10019-21ssh09:10:343
1453499117,3cyclictest13618-21grep08:15:143
1453499116,2cyclictest17448-21apt-get09:35:133
1453499115,2cyclictest225-21systemd-journal12:04:593
1453499110,10cyclictest29091-21ssh09:22:183
1453299119,1cyclictest31943-21tr07:45:142
1453299119,1cyclictest21042-21tr08:30:122
1453299119,1cyclictest13774-21tr08:15:152
1453299118,2cyclictest710-21grep08:55:162
1453299118,2cyclictest30977-21grep07:40:252
1453299118,2cyclictest29138-21head08:45:232
1453299118,2cyclictest25775-21grep08:40:142
1453299118,2cyclictest25083-21grep07:30:162
1453299118,2cyclictest18598-21smart_sda07:15:242
1453299118,2cyclictest18526-21grep08:25:142
1453299118,2cyclictest18093-21chrt08:24:462
1453299110,10cyclictest5141-21cut07:55:192
1453299110,10cyclictest11972-21grep09:50:262
1452899118,2cyclictest6136-21sh08:00:011
1452899118,2cyclictest5767-21sed09:05:161
1452899118,2cyclictest486-21head07:45:211
1452899118,2cyclictest3832-21wc07:55:001
1452899118,2cyclictest31914-21ssh11:20:091
1452899118,2cyclictest22409-21expr07:25:141
1452899118,2cyclictest20753-21sh12:10:571
1452899118,2cyclictest20654-21ssh09:36:221
1452899118,2cyclictest19652-21ssh09:16:281
1452899118,2cyclictest18644-21expr08:25:141
1452899118,2cyclictest17294-21sed11:10:251
1452899118,2cyclictest16783-21ls11:30:131
1452899117,3cyclictest1932-21tr09:25:201
1452899117,3cyclictest1932-21tr09:25:191
1452899110,10cyclictest0-21swapper/111:57:281
1452399119,1cyclictest2456-21grep11:40:260
1452399119,1cyclictest23531-21head10:55:150
1452399119,1cyclictest21281-21grep07:20:250
1452399118,2cyclictest31938-21tr07:45:140
1452399118,2cyclictest25001-21cstates11:35:140
1452399118,2cyclictest24518-21ssh09:20:050
1452399118,2cyclictest23837-21taskset07:26:590
1452399118,2cyclictest21019-21cut08:30:120
1452399118,2cyclictest20890-21ssh09:17:250
1452399118,2cyclictest17849-21grep07:15:140
1452399118,2cyclictest13422-21grep10:10:250
1452399118,2cyclictest11614-21grep12:05:250
1452399114,2cyclictest225-21systemd-journal08:42:030
1452399112,5cyclictest23130-21apache_accesses08:35:110
1453499108,1cyclictest58342sleep307:56:143
1453499108,1cyclictest22615-21taskset08:31:513
1453499108,1cyclictest18864-21expr08:25:143
1453499108,1cyclictest15987-21awk07:10:203
1453499107,2cyclictest9349-21grep08:05:153
1453499106,3cyclictest26289-21head08:40:173
1453499101,3cyclictest18677-21ssh09:55:133
1453299108,1cyclictest8757-21date08:05:012
1453299108,1cyclictest6767-21cat08:00:132
1453299107,2cyclictest5234-21tr09:05:132
1453299107,2cyclictest30548-21tr08:50:142
1453299107,2cyclictest23313-21tr07:25:222
1453299107,2cyclictest16084-21grep07:10:222
1453299100,4cyclictest225-21systemd-journal07:51:202
1452899108,1cyclictest31387-21taskset07:44:291
1452899108,1cyclictest25157-21taskset08:39:031
1452899108,1cyclictest10227-21cat08:05:251
1452899107,2cyclictest4692-21chrt09:04:351
1452899107,2cyclictest21675-21seq07:24:461
1452899107,2cyclictest21072-21expr08:30:121
1452899107,2cyclictest16409-21cat08:20:131
1452899102,3cyclictest29165-21chrt10:20:181
1452899100,9cyclictest275262sleep108:43:401
1452399108,1cyclictest1842-21grep07:50:120
1452399107,2cyclictest9583-21cat09:10:260
1452399107,2cyclictest32639-21ssh10:22:100
1452399107,2cyclictest24869-21ssh11:15:190
1452399107,2cyclictest16713-21grep08:20:150
1452399104,1cyclictest225-21systemd-journal08:54:030
145329997,1cyclictest2583-21chrt09:00:032
145329996,2cyclictest20345-21grep07:20:162
145239998,1cyclictest24354-21dpkg07:30:120
145239992,4cyclictest27631-21cron08:45:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional