You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-15 - 06:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jan 15, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2777221410,0sleep20-21swapper/222:19:392
2559021350,2sleep1558-21lldpd20:36:501
1455023933,2sleep30-21swapper/319:06:543
1456323428,2sleep10-21swapper/119:07:051
1488999331,28cyclictest224-21systemd-journal23:00:131
1488999302,27cyclictest7740-21diskmemload22:47:191
1488999301,26cyclictest558-21lldpd22:30:161
1488999300,29cyclictest26534-21ssh00:30:181
1488999300,28cyclictest0-21swapper/100:10:121
14892992927,2cyclictest0-21swapper/221:50:242
1488499290,1cyclictest141rcu_preempt23:59:520
1464822924,1sleep20-21swapper/219:08:102
1488999288,14cyclictest224-21systemd-journal21:32:471
1488999281,26cyclictest7744-21snmpd20:55:161
1488999281,21cyclictest30068-21ssh23:40:131
1488999280,27cyclictest0-21swapper/121:35:521
1488999280,27cyclictest0-21swapper/121:14:591
1488999280,24cyclictest224-21systemd-journal22:40:141
14897992711,2cyclictest39-21ksoftirqd/322:09:413
14892992724,2cyclictest32-21ksoftirqd/223:30:182
1488999274,18cyclictest224-21systemd-journal21:27:201
1488999273,23cyclictest7740-21diskmemload22:10:011
14889992722,3cyclictest23727-21wc00:00:271
1488999271,25cyclictest0-21swapper/100:07:491
1488999271,17cyclictest0-21swapper/122:52:311
1488999270,24cyclictest7740-21diskmemload22:06:151
1488999270,24cyclictest0-21swapper/122:57:571
1488999270,24cyclictest0-21swapper/122:57:561
1488999270,23cyclictest7740-21diskmemload21:54:491
1489799261,3cyclictest141rcu_preempt19:55:193
1489799260,3cyclictest224-21systemd-journal23:46:203
14892992621,3cyclictest32-21ksoftirqd/223:52:332
14892992621,1cyclictest32-21ksoftirqd/222:43:072
14892992619,3cyclictest32-21ksoftirqd/223:02:092
1488999261,18cyclictest224-21systemd-journal23:13:411
1488999260,24cyclictest0-21swapper/123:38:191
1488999260,23cyclictest7740-21diskmemload23:49:151
1488999260,23cyclictest0-21swapper/120:10:191
1488999260,22cyclictest0-21swapper/119:20:121
1488999260,20cyclictest0-21swapper/121:48:001
1488499261,20cyclictest224-21systemd-journal22:05:490
14892992515,6cyclictest32-21ksoftirqd/223:10:102
1488999253,21cyclictest558-21lldpd22:15:141
1488999251,23cyclictest0-21swapper/121:00:141
1488999251,23cyclictest0-21swapper/100:19:191
1488999251,22cyclictest7740-21diskmemload23:26:571
14889992510,14cyclictest500-21in:imuxsock23:30:091
14897992421,2cyclictest39-21ksoftirqd/322:30:223
14897992413,2cyclictest39-21ksoftirqd/321:13:483
1489299240,2cyclictest0-21swapper/221:56:522
1488999242,21cyclictest558-21lldpd21:55:111
1488999242,20cyclictest30645-21basename20:50:011
1488999240,23cyclictest0-21swapper/121:20:131
1488999240,23cyclictest0-21swapper/121:16:131
1488999240,21cyclictest30069-21mailstats22:20:221
1488999240,21cyclictest0-21swapper/120:20:151
14884992420,3cyclictest558-21lldpd23:43:160
1488499240,22cyclictest7744-21snmpd22:02:220
14897992318,4cyclictest39-21ksoftirqd/321:15:193
1489799231,5cyclictest30476-21sh23:15:053
1489299239,1cyclictest32-21ksoftirqd/220:49:192
1489299237,7cyclictest224-21systemd-journal19:30:002
1489299233,1cyclictest141rcu_preempt20:25:192
1489299232,19cyclictest0-21swapper/200:12:442
14892992317,6cyclictest3295-21kworker/u8:122:04:012
1489299231,3cyclictest558-21lldpd19:12:322
14892992313,4cyclictest141rcu_preempt21:05:122
14892992312,8cyclictest32-21ksoftirqd/200:25:002
14889992313,7cyclictest224-21systemd-journal22:00:101
1488999231,21cyclictest9542-21ssh23:50:121
1488999230,22cyclictest0-21swapper/123:05:061
1488499237,2cyclictest13-21ksoftirqd/022:51:120
1488499234,12cyclictest224-21systemd-journal21:05:530
1488499233,17cyclictest224-21systemd-journal22:27:590
14897992219,2cyclictest7740-21diskmemload23:51:053
14897992214,4cyclictest224-21systemd-journal00:03:053
1489799221,2cyclictest25596-21ls00:30:123
1489799221,20cyclictest558-21lldpd00:15:133
1489799220,4cyclictest141rcu_preempt23:43:483
1489299222,3cyclictest558-21lldpd23:26:222
14892992214,3cyclictest32-21ksoftirqd/219:45:152
14892992212,6cyclictest32-21ksoftirqd/222:55:142
14892992212,6cyclictest32-21ksoftirqd/222:55:132
1489299221,1cyclictest141rcu_preempt22:30:382
1489299220,3cyclictest141rcu_preempt22:36:582
1489299220,22cyclictest0-21swapper/223:18:032
1488999227,12cyclictest7740-21diskmemload21:40:111
1488999223,18cyclictest7744-21snmpd23:23:471
1488999222,19cyclictest558-21lldpd20:00:151
1488999221,20cyclictest14527-21grep20:15:151
1488999221,20cyclictest0-21swapper/119:55:161
1488999221,18cyclictest558-21lldpd23:15:221
1488999221,15cyclictest558-21lldpd19:40:001
1488999220,20cyclictest0-21swapper/122:28:041
1488999220,20cyclictest0-21swapper/100:21:541
14884992219,2cyclictest13-21ksoftirqd/000:06:370
1489799219,9cyclictest39-21ksoftirqd/320:38:133
1489799217,6cyclictest39-21ksoftirqd/322:15:053
1489799212,9cyclictest224-21systemd-journal22:27:543
14897992114,4cyclictest500-21in:imuxsock21:39:043
1489799210,2cyclictest3295-21kworker/u8:122:10:253
1489299218,1cyclictest32-21ksoftirqd/200:19:402
1489299216,1cyclictest7744-21snmpd00:09:262
1489299211,8cyclictest32-21ksoftirqd/220:20:282
1488999210,20cyclictest0-21swapper/123:55:221
1488499212,3cyclictest141rcu_preempt22:15:390
1488499212,1cyclictest141rcu_preempt00:10:410
1488499210,4cyclictest558-21lldpd23:02:250
1489799206,1cyclictest39-21ksoftirqd/323:27:053
1489799203,4cyclictest141rcu_preempt22:42:453
14897992018,1cyclictest7740-21diskmemload22:49:203
14897992017,2cyclictest39-21ksoftirqd/321:26:263
14897992016,1cyclictest39-21ksoftirqd/321:58:503
1489299208,3cyclictest141rcu_preempt23:22:502
1489299204,8cyclictest224-21systemd-journal00:37:322
1489299204,2cyclictest141rcu_preempt22:20:332
1489299203,5cyclictest21378-21apt-get20:30:212
1489299203,4cyclictest558-21lldpd00:33:082
1489299202,17cyclictest0-21swapper/222:25:352
14892992018,2cyclictest558-21lldpd23:44:252
14892992015,4cyclictest32-21ksoftirqd/221:19:592
1489299200,4cyclictest141rcu_preempt21:36:172
1488999204,15cyclictest558-21lldpd22:35:241
1488999202,16cyclictest7744-21snmpd19:40:021
1488999200,19cyclictest0-21swapper/121:05:171
1488999200,19cyclictest0-21swapper/100:27:461
1488499201,3cyclictest141rcu_preempt21:50:180
1488499200,1cyclictest141rcu_preempt23:22:390
1489799197,1cyclictest224-21systemd-journal00:28:573
1489799195,6cyclictest39-21ksoftirqd/322:50:503
1489799191,4cyclictest141rcu_preempt22:00:263
1489799190,5cyclictest224-21systemd-journal23:12:393
1489799190,2cyclictest224-21systemd-journal19:51:023
1489299193,2cyclictest141rcu_preempt21:25:292
14892991916,2cyclictest32-21ksoftirqd/221:20:442
14892991914,4cyclictest37850irq/131-snd_hda21:40:512
1489299190,11cyclictest0-21swapper/219:15:082
1488999195,6cyclictest224-21systemd-journal00:35:061
1488999195,11cyclictest7744-21snmpd19:10:141
1488999194,14cyclictest11315-21grep20:09:591
1488999193,15cyclictest7744-21snmpd20:25:161
14889991910,8cyclictest502-21rs:main1
1488999190,18cyclictest0-21swapper/119:50:151
1488499196,2cyclictest7744-21snmpd22:22:460
1488499192,1cyclictest141rcu_preempt22:12:370
14884991916,2cyclictest13-21ksoftirqd/022:50:000
1488499191,4cyclictest7740-21diskmemload21:12:520
14884991914,4cyclictest13-21ksoftirqd/023:38:100
14884991912,3cyclictest13-21ksoftirqd/021:35:230
1488499190,1cyclictest141rcu_preempt23:46:380
1488499190,1cyclictest141rcu_preempt22:30:300
44932180,0sleep30-21swapper/300:11:233
1489799186,8cyclictest39-21ksoftirqd/323:31:463
1489799186,7cyclictest39-21ksoftirqd/322:22:263
1489799183,1cyclictest39-21ksoftirqd/321:45:573
14897991815,2cyclictest558-21lldpd22:37:433
1489799181,1cyclictest141rcu_preempt20:34:183
14897991810,4cyclictest19747-21grep20:25:163
14897991810,2cyclictest39-21ksoftirqd/321:00:153
1489299182,12cyclictest224-21systemd-journal23:08:552
14892991816,1cyclictest558-21lldpd22:12:192
14892991814,3cyclictest32-21ksoftirqd/221:00:002
14892991814,3cyclictest32-21ksoftirqd/220:10:192
14892991813,4cyclictest32-21ksoftirqd/200:26:052
14892991812,3cyclictest32-21ksoftirqd/222:50:082
1489299180,1cyclictest0-21swapper/200:02:152
1488999182,7cyclictest224-21systemd-journal20:40:141
1488999181,16cyclictest558-21lldpd19:30:171
1488999181,14cyclictest224-21systemd-journal20:30:201
1488999180,16cyclictest0-21swapper/119:25:021
1488499189,6cyclictest16409-21apache_processe23:30:090
1488499180,3cyclictest141rcu_preempt21:58:070
1488499180,2cyclictest141rcu_preempt22:55:410
1488499180,2cyclictest141rcu_preempt22:55:410
1489799177,4cyclictest39-21ksoftirqd/321:33:503
1489799173,7cyclictest224-21systemd-journal21:53:043
1489799172,7cyclictest224-21systemd-journal21:24:403
14897991714,2cyclictest39-21ksoftirqd/320:25:013
14897991713,3cyclictest39-21ksoftirqd/323:55:163
1489299176,5cyclictest558-21lldpd23:35:162
1489299173,1cyclictest141rcu_preempt19:36:192
14892991713,3cyclictest32-21ksoftirqd/223:45:272
14892991712,2cyclictest7744-21snmpd20:41:502
14892991710,2cyclictest32-21ksoftirqd/221:30:152
1489299170,5cyclictest32-21ksoftirqd/220:15:162
1488999174,12cyclictest7744-21snmpd20:50:241
1488499179,7cyclictest558-21lldpd00:24:200
1488499176,1cyclictest141rcu_preempt00:15:550
1488499174,3cyclictest9901-21kworker/u8:023:05:430
1488499174,1cyclictest141rcu_preempt00:35:140
1488499173,3cyclictest141rcu_preempt21:00:000
1488499172,2cyclictest13-21ksoftirqd/020:53:120
1488499170,1cyclictest558-21lldpd19:30:340
1488499170,1cyclictest141rcu_preempt00:31:100
1489799169,4cyclictest39-21ksoftirqd/320:00:183
1489799166,4cyclictest39-21ksoftirqd/320:50:193
1489799165,6cyclictest39-21ksoftirqd/319:19:403
1489799165,10cyclictest39-21ksoftirqd/300:20:213
1489799164,6cyclictest224-21systemd-journal00:07:573
1489799163,8cyclictest224-21systemd-journal19:25:193
1489799162,1cyclictest0-21swapper/319:40:393
14897991614,2cyclictest39-21ksoftirqd/323:35:313
14897991613,2cyclictest39-21ksoftirqd/323:02:093
1489799161,1cyclictest141rcu_preempt20:48:593
14897991610,2cyclictest39-21ksoftirqd/322:56:193
14897991610,2cyclictest39-21ksoftirqd/322:56:193
1489299168,6cyclictest500-21in:imuxsock23:59:102
1489299166,7cyclictest7744-21snmpd21:10:262
1489299163,3cyclictest32-21ksoftirqd/221:48:402
14892991613,2cyclictest32-21ksoftirqd/222:06:432
1488999162,13cyclictest558-21lldpd19:45:191
1488499168,1cyclictest29948-21ssh23:14:440
1488499163,3cyclictest141rcu_preempt00:00:300
1488499162,3cyclictest13-21ksoftirqd/020:03:490
14884991613,2cyclictest7744-21snmpd21:22:250
1488499160,2cyclictest141rcu_preempt00:29:020
1489799155,4cyclictest39-21ksoftirqd/323:23:503
1489799155,4cyclictest39-21ksoftirqd/323:05:413
1489799155,4cyclictest39-21ksoftirqd/321:40:143
1489799154,4cyclictest224-21systemd-journal00:35:173
1489799150,1cyclictest141rcu_preempt20:44:133
1489299152,4cyclictest20086-21cut19:20:142
1489299152,2cyclictest9176-21apache_processe20:05:132
1489299151,5cyclictest558-21lldpd19:50:152
1489299150,12cyclictest28990-21ssh22:46:462
1488499159,2cyclictest13-21ksoftirqd/021:26:160
1488499158,3cyclictest13-21ksoftirqd/023:15:040
1488499155,2cyclictest224-21systemd-journal20:19:370
1488499153,5cyclictest7744-21snmpd23:25:500
1488499152,1cyclictest141rcu_preempt20:39:030
14884991514,1cyclictest13-21ksoftirqd/021:46:410
14884991513,1cyclictest7744-21snmpd21:34:460
1488499151,2cyclictest141rcu_preempt22:40:310
1488499150,8cyclictest21116-21idleruntime-cro20:30:000
1488499150,2cyclictest13-21ksoftirqd/019:20:200
1489799149,4cyclictest39-21ksoftirqd/319:20:143
1489799147,4cyclictest39-21ksoftirqd/320:06:313
1489799147,1cyclictest39-21ksoftirqd/319:35:173
1489799146,1cyclictest39-21ksoftirqd/320:12:353
1489799145,1cyclictest39-21ksoftirqd/320:15:113
1489799145,1cyclictest39-21ksoftirqd/319:45:203
1489799144,4cyclictest39-21ksoftirqd/320:55:223
1489799144,1cyclictest39-21ksoftirqd/319:10:013
1489799143,1cyclictest39-21ksoftirqd/321:07:013
1489299142,4cyclictest558-21lldpd21:05:002
1489299142,2cyclictest141rcu_preempt20:00:342
14892991411,3cyclictest29387-21kworker/u8:120:50:132
1488499148,5cyclictest291692sleep019:37:060
1488499145,3cyclictest13-21ksoftirqd/021:16:550
1488499143,9cyclictest141rcu_preempt23:53:470
1488499142,7cyclictest141rcu_preempt20:24:360
1488499141,7cyclictest141rcu_preempt19:14:330
1488499141,4cyclictest141rcu_preempt20:43:250
1488499141,1cyclictest558-21lldpd21:44:020
14884991410,3cyclictest13-21ksoftirqd/019:25:000
1488499140,8cyclictest0-21swapper/020:30:580
1488499140,8cyclictest0-21swapper/019:53:530
1488499140,2cyclictest141rcu_preempt19:18:420
1489799131,11cyclictest25023-21aten2_r7power_p19:30:183
1489299138,1cyclictest32-21ksoftirqd/219:40:202
1489299133,1cyclictest141rcu_preempt19:58:282
14892991311,2cyclictest37850irq/131-snd_hda20:36:172
14892991310,2cyclictest32-21ksoftirqd/219:30:182
1488499135,2cyclictest224-21systemd-journal20:08:060
1488499134,3cyclictest224-21systemd-journal20:47:420
1488499134,3cyclictest141rcu_preempt19:40:380
1488499132,1cyclictest13-21ksoftirqd/019:55:080
1488499131,8cyclictest141rcu_preempt20:12:130
1488499131,3cyclictest558-21lldpd22:35:500
1488499130,4cyclictest0-21swapper/019:48:200
1488499124,2cyclictest224-21systemd-journal21:00:220
145572113,4sleep00-21swapper/019:07:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional